2024-09-17T06:58:43.527 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-17T06:58:43.543 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T06:58:43.625 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi/7908141 branch: reef description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/classic} email: ceph-qa@ceph.com first_in_suite: false job_id: '7908141' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: reef ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: true mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - CEPHADM_STRAY_DAEMON - CEPHADM_FAILED_DAEMON - CEPHADM_AGENT_DOWN log-only-match: - CEPHADM_ sha1: 19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 workunit: branch: reef sha1: 1cbdc9211df088cccbd94423c7cf3a2b85fe432b owner: scheduled_teuthology@teuthology priority: 930 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 5234 sha1: 19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 sleep_before_teardown: 0 subset: 46/64 suite: orch suite_branch: reef suite_path: /home/teuthworker/src/git.ceph.com_ceph_1cbdc9211df088cccbd94423c7cf3a2b85fe432b/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 1cbdc9211df088cccbd94423c7cf3a2b85fe432b targets: smithi044.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKhliw/a1AIeNdklEppdcpxWsbFK8dvECONzTcorqnInTMtfUqg3mQbYFVwQJE4t0GDPi6VuXHqtn+cfWcGFTuE= smithi179.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIjAE5rOEWC6ycPnRrxvpmPPnPGyXMh0xcrx4x1t0edqn1PYrHA75nS014GS4fzmHWHMIQ1M1+nXPkST2bwsrpQ= tasks: - cephadm: allow_ptrace: false avoid_pacific_features: true cephadm_branch: v16.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v16.2.0 - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph orch apply rgw smpl - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 120 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-16_22:08:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-17T06:58:43.625 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_1cbdc9211df088cccbd94423c7cf3a2b85fe432b/qa; will attempt to use it 2024-09-17T06:58:43.626 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_1cbdc9211df088cccbd94423c7cf3a2b85fe432b/qa/tasks 2024-09-17T06:58:43.626 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-17T06:58:43.628 INFO:teuthology.task.internal:Checking packages... 2024-09-17T06:58:43.646 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9' 2024-09-17T06:58:43.647 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-17T06:58:43.647 INFO:teuthology.packaging:ref: None 2024-09-17T06:58:43.647 INFO:teuthology.packaging:tag: None 2024-09-17T06:58:43.647 INFO:teuthology.packaging:branch: reef 2024-09-17T06:58:43.647 INFO:teuthology.packaging:sha1: 19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T06:58:43.647 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-09-17T06:58:43.940 INFO:teuthology.task.internal:Found packages for ceph version 18.2.4-855.g19e2a2be 2024-09-17T06:58:43.942 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-17T06:58:43.949 INFO:teuthology.task.internal:no buildpackages task found 2024-09-17T06:58:43.949 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-17T06:58:43.961 INFO:teuthology.task.internal:Saving configuration 2024-09-17T06:58:43.972 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-17T06:58:43.979 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-17T06:58:44.004 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi044.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi/7908141', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 06:45:37.212769', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKhliw/a1AIeNdklEppdcpxWsbFK8dvECONzTcorqnInTMtfUqg3mQbYFVwQJE4t0GDPi6VuXHqtn+cfWcGFTuE='} 2024-09-17T06:58:44.028 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi179.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi/7908141', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 06:45:37.214138', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIjAE5rOEWC6ycPnRrxvpmPPnPGyXMh0xcrx4x1t0edqn1PYrHA75nS014GS4fzmHWHMIQ1M1+nXPkST2bwsrpQ='} 2024-09-17T06:58:44.028 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-17T06:58:44.035 INFO:teuthology.task.internal:roles: ubuntu@smithi044.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-09-17T06:58:44.035 INFO:teuthology.task.internal:roles: ubuntu@smithi179.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-09-17T06:58:44.035 INFO:teuthology.run_tasks:Running task console_log... 2024-09-17T06:58:44.095 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc66bac3ac0>, signals=[15]) 2024-09-17T06:58:44.095 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-17T06:58:44.101 INFO:teuthology.task.internal:Opening connections... 2024-09-17T06:58:44.101 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi044.front.sepia.ceph.com 2024-09-17T06:58:44.102 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T06:58:44.179 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi179.front.sepia.ceph.com 2024-09-17T06:58:44.180 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T06:58:44.255 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-17T06:58:44.261 DEBUG:teuthology.orchestra.run.smithi044:> uname -m 2024-09-17T06:58:44.280 INFO:teuthology.orchestra.run.smithi044.stdout:x86_64 2024-09-17T06:58:44.280 DEBUG:teuthology.orchestra.run.smithi044:> cat /etc/os-release 2024-09-17T06:58:44.337 INFO:teuthology.orchestra.run.smithi044.stdout:NAME="CentOS Stream" 2024-09-17T06:58:44.338 INFO:teuthology.orchestra.run.smithi044.stdout:VERSION="9" 2024-09-17T06:58:44.338 INFO:teuthology.orchestra.run.smithi044.stdout:ID="centos" 2024-09-17T06:58:44.338 INFO:teuthology.orchestra.run.smithi044.stdout:ID_LIKE="rhel fedora" 2024-09-17T06:58:44.338 INFO:teuthology.orchestra.run.smithi044.stdout:VERSION_ID="9" 2024-09-17T06:58:44.338 INFO:teuthology.orchestra.run.smithi044.stdout:PLATFORM_ID="platform:el9" 2024-09-17T06:58:44.338 INFO:teuthology.orchestra.run.smithi044.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T06:58:44.338 INFO:teuthology.orchestra.run.smithi044.stdout:ANSI_COLOR="0;31" 2024-09-17T06:58:44.338 INFO:teuthology.orchestra.run.smithi044.stdout:LOGO="fedora-logo-icon" 2024-09-17T06:58:44.338 INFO:teuthology.orchestra.run.smithi044.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T06:58:44.338 INFO:teuthology.orchestra.run.smithi044.stdout:HOME_URL="https://centos.org/" 2024-09-17T06:58:44.338 INFO:teuthology.orchestra.run.smithi044.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T06:58:44.338 INFO:teuthology.orchestra.run.smithi044.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T06:58:44.339 INFO:teuthology.orchestra.run.smithi044.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T06:58:44.339 INFO:teuthology.lock.ops:Updating smithi044.front.sepia.ceph.com on lock server 2024-09-17T06:58:44.362 DEBUG:teuthology.orchestra.run.smithi179:> uname -m 2024-09-17T06:58:44.380 INFO:teuthology.orchestra.run.smithi179.stdout:x86_64 2024-09-17T06:58:44.381 DEBUG:teuthology.orchestra.run.smithi179:> cat /etc/os-release 2024-09-17T06:58:44.438 INFO:teuthology.orchestra.run.smithi179.stdout:NAME="CentOS Stream" 2024-09-17T06:58:44.438 INFO:teuthology.orchestra.run.smithi179.stdout:VERSION="9" 2024-09-17T06:58:44.438 INFO:teuthology.orchestra.run.smithi179.stdout:ID="centos" 2024-09-17T06:58:44.438 INFO:teuthology.orchestra.run.smithi179.stdout:ID_LIKE="rhel fedora" 2024-09-17T06:58:44.439 INFO:teuthology.orchestra.run.smithi179.stdout:VERSION_ID="9" 2024-09-17T06:58:44.439 INFO:teuthology.orchestra.run.smithi179.stdout:PLATFORM_ID="platform:el9" 2024-09-17T06:58:44.439 INFO:teuthology.orchestra.run.smithi179.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T06:58:44.439 INFO:teuthology.orchestra.run.smithi179.stdout:ANSI_COLOR="0;31" 2024-09-17T06:58:44.439 INFO:teuthology.orchestra.run.smithi179.stdout:LOGO="fedora-logo-icon" 2024-09-17T06:58:44.439 INFO:teuthology.orchestra.run.smithi179.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T06:58:44.439 INFO:teuthology.orchestra.run.smithi179.stdout:HOME_URL="https://centos.org/" 2024-09-17T06:58:44.439 INFO:teuthology.orchestra.run.smithi179.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T06:58:44.440 INFO:teuthology.orchestra.run.smithi179.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T06:58:44.440 INFO:teuthology.orchestra.run.smithi179.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T06:58:44.440 INFO:teuthology.lock.ops:Updating smithi179.front.sepia.ceph.com on lock server 2024-09-17T06:58:44.461 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-17T06:58:44.470 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-17T06:58:44.476 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-17T06:58:44.476 DEBUG:teuthology.orchestra.run.smithi044:> test '!' -e /home/ubuntu/cephtest 2024-09-17T06:58:44.479 DEBUG:teuthology.orchestra.run.smithi179:> test '!' -e /home/ubuntu/cephtest 2024-09-17T06:58:44.495 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-17T06:58:44.501 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-17T06:58:44.501 DEBUG:teuthology.orchestra.run.smithi044:> test -z $(ls -A /var/lib/ceph) 2024-09-17T06:58:44.537 DEBUG:teuthology.orchestra.run.smithi179:> test -z $(ls -A /var/lib/ceph) 2024-09-17T06:58:44.562 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-17T06:58:44.650 INFO:teuthology.run_tasks:Running task kernel... 2024-09-17T06:58:44.664 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-17T06:58:44.664 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-17T06:58:44.664 DEBUG:teuthology.orchestra.run.smithi044:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T06:58:44.665 DEBUG:teuthology.orchestra.run.smithi179:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T06:58:44.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T06:58:44.681 DEBUG:teuthology.orchestra.run.smithi179:> uname -r 2024-09-17T06:58:44.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T06:58:44.682 DEBUG:teuthology.orchestra.run.smithi044:> uname -r 2024-09-17T06:58:44.738 INFO:teuthology.orchestra.run.smithi179.stdout:5.14.0-503.el9.x86_64 2024-09-17T06:58:44.739 INFO:teuthology.task.kernel:Running kernel on smithi179: 5.14.0-503.el9.x86_64 2024-09-17T06:58:44.739 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum install -y kernel 2024-09-17T06:58:44.740 INFO:teuthology.orchestra.run.smithi044.stdout:5.14.0-503.el9.x86_64 2024-09-17T06:58:44.740 INFO:teuthology.task.kernel:Running kernel on smithi044: 5.14.0-503.el9.x86_64 2024-09-17T06:58:44.740 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum install -y kernel 2024-09-17T06:58:46.998 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.3 MB 00:00 2024-09-17T06:58:47.090 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.3 MB 00:00 2024-09-17T06:58:48.809 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - AppStream 31 MB/s | 20 MB 00:00 2024-09-17T06:58:49.016 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - AppStream 22 MB/s | 20 MB 00:00 2024-09-17T06:58:53.193 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-17T06:58:54.467 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - CRB 4.3 MB/s | 6.5 MB 00:01 2024-09-17T06:58:54.893 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - Extras packages 48 kB/s | 19 kB 00:00 2024-09-17T06:58:56.005 INFO:teuthology.orchestra.run.smithi179.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-17T06:58:56.034 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - Extras packages 61 kB/s | 19 kB 00:00 2024-09-17T06:58:57.038 INFO:teuthology.orchestra.run.smithi044.stdout:Extra Packages for Enterprise Linux 26 MB/s | 23 MB 00:00 2024-09-17T06:59:01.412 INFO:teuthology.orchestra.run.smithi179.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-17T06:59:02.669 INFO:teuthology.orchestra.run.smithi044.stdout:lab-extras 21 kB/s | 1.7 kB 00:00 2024-09-17T06:59:03.318 INFO:teuthology.orchestra.run.smithi179.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-17T06:59:03.318 INFO:teuthology.orchestra.run.smithi179.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-17T06:59:03.403 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-09-17T06:59:03.428 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-09-17T06:59:03.428 INFO:teuthology.orchestra.run.smithi179.stdout: Package Architecture Version Repository Size 2024-09-17T06:59:03.428 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-09-17T06:59:03.428 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-09-17T06:59:03.428 INFO:teuthology.orchestra.run.smithi179.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-17T06:59:03.428 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-09-17T06:59:03.428 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-17T06:59:03.428 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-17T06:59:03.428 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-17T06:59:03.428 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-09-17T06:59:03.429 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-09-17T06:59:03.429 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-09-17T06:59:03.429 INFO:teuthology.orchestra.run.smithi179.stdout:Install 4 Packages 2024-09-17T06:59:03.429 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-09-17T06:59:03.430 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 79 M 2024-09-17T06:59:03.430 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 126 M 2024-09-17T06:59:03.430 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-09-17T06:59:03.763 INFO:teuthology.orchestra.run.smithi179.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 468 kB/s | 36 kB 00:00 2024-09-17T06:59:04.355 INFO:teuthology.orchestra.run.smithi179.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 23 MB/s | 16 MB 00:00 2024-09-17T06:59:04.634 INFO:teuthology.orchestra.run.smithi044.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-17T06:59:04.635 INFO:teuthology.orchestra.run.smithi044.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-17T06:59:04.737 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-09-17T06:59:04.747 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-09-17T06:59:04.748 INFO:teuthology.orchestra.run.smithi044.stdout: Package Architecture Version Repository Size 2024-09-17T06:59:04.748 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-09-17T06:59:04.748 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-09-17T06:59:04.748 INFO:teuthology.orchestra.run.smithi044.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-17T06:59:04.748 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-09-17T06:59:04.748 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-17T06:59:04.748 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-17T06:59:04.748 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-17T06:59:04.748 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-09-17T06:59:04.748 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-09-17T06:59:04.748 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-09-17T06:59:04.748 INFO:teuthology.orchestra.run.smithi044.stdout:Install 4 Packages 2024-09-17T06:59:04.748 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-09-17T06:59:04.749 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 79 M 2024-09-17T06:59:04.749 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 126 M 2024-09-17T06:59:04.749 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-09-17T06:59:05.030 INFO:teuthology.orchestra.run.smithi179.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 26 MB/s | 35 MB 00:01 2024-09-17T06:59:05.129 INFO:teuthology.orchestra.run.smithi044.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 355 kB/s | 36 kB 00:00 2024-09-17T06:59:05.322 INFO:teuthology.orchestra.run.smithi179.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 18 MB/s | 29 MB 00:01 2024-09-17T06:59:05.322 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:59:05.322 INFO:teuthology.orchestra.run.smithi179.stdout:Total 42 MB/s | 79 MB 00:01 2024-09-17T06:59:05.521 INFO:teuthology.orchestra.run.smithi044.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 32 MB/s | 16 MB 00:00 2024-09-17T06:59:05.996 INFO:teuthology.orchestra.run.smithi044.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 33 MB/s | 29 MB 00:00 2024-09-17T06:59:06.000 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-09-17T06:59:06.305 INFO:teuthology.orchestra.run.smithi044.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 27 MB/s | 35 MB 00:01 2024-09-17T06:59:06.305 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-09-17T06:59:06.305 INFO:teuthology.orchestra.run.smithi044.stdout:Total 51 MB/s | 79 MB 00:01 2024-09-17T06:59:06.349 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-09-17T06:59:06.349 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-09-17T06:59:07.021 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-09-17T06:59:07.367 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-09-17T06:59:07.367 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-09-17T06:59:07.417 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-09-17T06:59:07.417 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-09-17T06:59:08.415 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-09-17T06:59:08.416 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-09-17T06:59:09.808 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-09-17T06:59:10.455 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-17T06:59:10.539 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T06:59:10.995 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-09-17T06:59:11.669 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-17T06:59:11.749 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T06:59:13.314 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T06:59:13.441 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T06:59:14.522 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T06:59:14.652 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T06:59:18.262 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T06:59:18.308 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T06:59:19.782 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T06:59:19.829 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T06:59:23.171 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T06:59:24.966 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T07:00:01.366 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T07:00:01.406 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-17T07:00:01.725 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T07:00:01.725 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-17T07:00:01.725 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T07:00:01.725 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T07:00:02.597 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T07:00:02.597 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-09-17T07:00:02.598 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-09-17T07:00:02.598 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-17T07:00:02.598 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-17T07:00:02.598 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-17T07:00:02.598 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-17T07:00:02.598 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-09-17T07:00:02.598 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-09-17T07:00:02.973 DEBUG:teuthology.orchestra.run.smithi179:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T07:00:03.064 INFO:teuthology.orchestra.run.smithi179.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T07:00:03.065 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T07:00:03.065 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-17T07:00:03.065 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T07:00:03.065 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T07:00:03.065 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-17T07:00:03.065 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi179.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T07:00:03.065 DEBUG:teuthology.orchestra.run.smithi179:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T07:00:03.125 INFO:teuthology.orchestra.run.smithi179.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T07:00:03.125 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T07:00:03.125 DEBUG:teuthology.orchestra.run.smithi179:> sudo rpm -qi grub2-tools 2024-09-17T07:00:03.172 INFO:teuthology.orchestra.run.smithi179.stdout:Name : grub2-tools 2024-09-17T07:00:03.172 INFO:teuthology.orchestra.run.smithi179.stdout:Epoch : 1 2024-09-17T07:00:03.172 INFO:teuthology.orchestra.run.smithi179.stdout:Version : 2.06 2024-09-17T07:00:03.172 INFO:teuthology.orchestra.run.smithi179.stdout:Release : 82.el9 2024-09-17T07:00:03.173 INFO:teuthology.orchestra.run.smithi179.stdout:Architecture: x86_64 2024-09-17T07:00:03.173 INFO:teuthology.orchestra.run.smithi179.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-17T07:00:03.173 INFO:teuthology.orchestra.run.smithi179.stdout:Group : Unspecified 2024-09-17T07:00:03.173 INFO:teuthology.orchestra.run.smithi179.stdout:Size : 8274670 2024-09-17T07:00:03.173 INFO:teuthology.orchestra.run.smithi179.stdout:License : GPLv3+ 2024-09-17T07:00:03.173 INFO:teuthology.orchestra.run.smithi179.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-17T07:00:03.173 INFO:teuthology.orchestra.run.smithi179.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-17T07:00:03.173 INFO:teuthology.orchestra.run.smithi179.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-17T07:00:03.173 INFO:teuthology.orchestra.run.smithi179.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-17T07:00:03.173 INFO:teuthology.orchestra.run.smithi179.stdout:Packager : builder@centos.org 2024-09-17T07:00:03.173 INFO:teuthology.orchestra.run.smithi179.stdout:Vendor : CentOS 2024-09-17T07:00:03.173 INFO:teuthology.orchestra.run.smithi179.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-17T07:00:03.173 INFO:teuthology.orchestra.run.smithi179.stdout:Summary : Support tools for GRUB. 2024-09-17T07:00:03.174 INFO:teuthology.orchestra.run.smithi179.stdout:Description : 2024-09-17T07:00:03.174 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-09-17T07:00:03.174 INFO:teuthology.orchestra.run.smithi179.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-17T07:00:03.174 INFO:teuthology.orchestra.run.smithi179.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-17T07:00:03.174 INFO:teuthology.orchestra.run.smithi179.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-17T07:00:03.174 INFO:teuthology.orchestra.run.smithi179.stdout:hardware devices. 2024-09-17T07:00:03.174 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-09-17T07:00:03.174 INFO:teuthology.orchestra.run.smithi179.stdout:This subpackage provides tools for support of all platforms. 2024-09-17T07:00:03.175 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-17T07:00:03.175 INFO:teuthology.task.kernel:Updating grub on smithi179 to boot 5.14.0-508.el9.x86_64 2024-09-17T07:00:03.175 DEBUG:teuthology.orchestra.run.smithi179:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-17T07:00:03.833 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T07:00:03.852 INFO:teuthology.orchestra.run.smithi179.stderr:Generating grub configuration file ... 2024-09-17T07:00:03.875 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-17T07:00:04.236 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T07:00:04.237 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-17T07:00:04.237 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T07:00:04.237 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T07:00:05.106 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T07:00:05.107 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-09-17T07:00:05.107 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-09-17T07:00:05.107 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-17T07:00:05.107 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-17T07:00:05.107 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-17T07:00:05.107 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-17T07:00:05.107 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-09-17T07:00:05.107 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-09-17T07:00:05.210 INFO:teuthology.orchestra.run.smithi179.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-17T07:00:05.238 INFO:teuthology.orchestra.run.smithi179.stderr:done 2024-09-17T07:00:05.240 DEBUG:teuthology.orchestra.run.smithi179:> mktemp 2024-09-17T07:00:05.256 INFO:teuthology.orchestra.run.smithi179.stdout:/tmp/tmp.LCbkuYtMVm 2024-09-17T07:00:05.256 DEBUG:teuthology.orchestra.run.smithi179:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.LCbkuYtMVm 2024-09-17T07:00:05.323 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 0666 /tmp/tmp.LCbkuYtMVm 2024-09-17T07:00:05.452 DEBUG:teuthology.orchestra.run.smithi044:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T07:00:05.479 DEBUG:teuthology.orchestra.remote:smithi179:/tmp/tmp.LCbkuYtMVm is 6KB 2024-09-17T07:00:05.527 DEBUG:teuthology.orchestra.run.smithi179:> rm -fr /tmp/tmp.LCbkuYtMVm 2024-09-17T07:00:05.542 DEBUG:teuthology.orchestra.run.smithi179:> sudo /bin/ls /boot/loader/entries || true 2024-09-17T07:00:05.544 INFO:teuthology.orchestra.run.smithi044.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T07:00:05.545 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T07:00:05.545 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-17T07:00:05.545 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T07:00:05.545 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T07:00:05.545 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-17T07:00:05.545 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi044.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T07:00:05.545 DEBUG:teuthology.orchestra.run.smithi044:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T07:00:05.604 INFO:teuthology.orchestra.run.smithi044.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T07:00:05.605 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T07:00:05.605 DEBUG:teuthology.orchestra.run.smithi044:> sudo rpm -qi grub2-tools 2024-09-17T07:00:05.608 INFO:teuthology.orchestra.run.smithi179.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-17T07:00:05.608 INFO:teuthology.orchestra.run.smithi179.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-17T07:00:05.608 INFO:teuthology.orchestra.run.smithi179.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-17T07:00:05.608 INFO:teuthology.orchestra.run.smithi179.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-17T07:00:05.609 DEBUG:teuthology.orchestra.run.smithi179:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-17T07:00:05.642 INFO:teuthology.orchestra.run.smithi044.stdout:Name : grub2-tools 2024-09-17T07:00:05.642 INFO:teuthology.orchestra.run.smithi044.stdout:Epoch : 1 2024-09-17T07:00:05.642 INFO:teuthology.orchestra.run.smithi044.stdout:Version : 2.06 2024-09-17T07:00:05.642 INFO:teuthology.orchestra.run.smithi044.stdout:Release : 82.el9 2024-09-17T07:00:05.642 INFO:teuthology.orchestra.run.smithi044.stdout:Architecture: x86_64 2024-09-17T07:00:05.642 INFO:teuthology.orchestra.run.smithi044.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-17T07:00:05.642 INFO:teuthology.orchestra.run.smithi044.stdout:Group : Unspecified 2024-09-17T07:00:05.643 INFO:teuthology.orchestra.run.smithi044.stdout:Size : 8274670 2024-09-17T07:00:05.643 INFO:teuthology.orchestra.run.smithi044.stdout:License : GPLv3+ 2024-09-17T07:00:05.643 INFO:teuthology.orchestra.run.smithi044.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-17T07:00:05.643 INFO:teuthology.orchestra.run.smithi044.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-17T07:00:05.643 INFO:teuthology.orchestra.run.smithi044.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-17T07:00:05.643 INFO:teuthology.orchestra.run.smithi044.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-17T07:00:05.643 INFO:teuthology.orchestra.run.smithi044.stdout:Packager : builder@centos.org 2024-09-17T07:00:05.643 INFO:teuthology.orchestra.run.smithi044.stdout:Vendor : CentOS 2024-09-17T07:00:05.643 INFO:teuthology.orchestra.run.smithi044.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-17T07:00:05.643 INFO:teuthology.orchestra.run.smithi044.stdout:Summary : Support tools for GRUB. 2024-09-17T07:00:05.643 INFO:teuthology.orchestra.run.smithi044.stdout:Description : 2024-09-17T07:00:05.643 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-09-17T07:00:05.643 INFO:teuthology.orchestra.run.smithi044.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-17T07:00:05.643 INFO:teuthology.orchestra.run.smithi044.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-17T07:00:05.644 INFO:teuthology.orchestra.run.smithi044.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-17T07:00:05.644 INFO:teuthology.orchestra.run.smithi044.stdout:hardware devices. 2024-09-17T07:00:05.644 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-09-17T07:00:05.644 INFO:teuthology.orchestra.run.smithi044.stdout:This subpackage provides tools for support of all platforms. 2024-09-17T07:00:05.645 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-17T07:00:05.645 INFO:teuthology.task.kernel:Updating grub on smithi044 to boot 5.14.0-508.el9.x86_64 2024-09-17T07:00:05.645 DEBUG:teuthology.orchestra.run.smithi044:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-17T07:00:05.759 DEBUG:teuthology.orchestra.run.smithi179:> sudo shutdown -r now 2024-09-17T07:00:06.289 INFO:teuthology.orchestra.run.smithi044.stderr:Generating grub configuration file ... 2024-09-17T07:00:07.555 INFO:teuthology.orchestra.run.smithi044.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-17T07:00:07.584 INFO:teuthology.orchestra.run.smithi044.stderr:done 2024-09-17T07:00:07.589 DEBUG:teuthology.orchestra.run.smithi044:> mktemp 2024-09-17T07:00:07.619 INFO:teuthology.orchestra.run.smithi044.stdout:/tmp/tmp.dKRpzi2a28 2024-09-17T07:00:07.619 DEBUG:teuthology.orchestra.run.smithi044:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.dKRpzi2a28 2024-09-17T07:00:07.685 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 0666 /tmp/tmp.dKRpzi2a28 2024-09-17T07:00:07.836 DEBUG:teuthology.orchestra.remote:smithi044:/tmp/tmp.dKRpzi2a28 is 6KB 2024-09-17T07:00:07.884 DEBUG:teuthology.orchestra.run.smithi044:> rm -fr /tmp/tmp.dKRpzi2a28 2024-09-17T07:00:07.899 DEBUG:teuthology.orchestra.run.smithi044:> sudo /bin/ls /boot/loader/entries || true 2024-09-17T07:00:07.971 INFO:teuthology.orchestra.run.smithi044.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-17T07:00:07.971 INFO:teuthology.orchestra.run.smithi044.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-17T07:00:07.971 INFO:teuthology.orchestra.run.smithi044.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-17T07:00:07.971 INFO:teuthology.orchestra.run.smithi044.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-17T07:00:07.972 DEBUG:teuthology.orchestra.run.smithi044:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-17T07:00:08.132 DEBUG:teuthology.orchestra.run.smithi044:> sudo shutdown -r now 2024-09-17T07:00:35.790 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-17T07:00:35.790 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-09-17T07:00:35.791 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T07:00:38.136 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-17T07:00:38.136 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-09-17T07:00:38.137 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T07:00:51.249 ERROR:teuthology.orchestra.connection:Error authenticating with smithi179.front.sepia.ceph.com: AuthenticationException('Authentication failed.') 2024-09-17T07:00:55.321 ERROR:teuthology.orchestra.connection:Error authenticating with smithi179.front.sepia.ceph.com: AuthenticationException('Authentication failed.') 2024-09-17T07:00:56.513 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.44 2024-09-17T07:01:02.393 ERROR:teuthology.orchestra.connection:Error authenticating with smithi179.front.sepia.ceph.com: AuthenticationException('Authentication failed.') 2024-09-17T07:01:05.514 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-09-17T07:01:05.514 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T07:01:27.848 ERROR:teuthology.orchestra.connection:Error authenticating with smithi179.front.sepia.ceph.com: AuthenticationException('Authentication failed.') 2024-09-17T07:01:40.888 ERROR:teuthology.orchestra.connection:Error authenticating with smithi179.front.sepia.ceph.com: BadHostKeyException('smithi179.front.sepia.ceph.com', PKey(alg=ECDSA, bits=256, fp=SHA256:62YHXZ/hIv7z5T30+I7hwDbCw/enRJjcemfW9DQTOfg), PKey(alg=ED25519, bits=256, fp=SHA256:W+6Y/LgnRby4510vqEHFE/6fAReT8MdM2qNnjG5ecjQ)) (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/connection.py", line 111, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 468, in connect raise BadHostKeyException(hostname, server_key, our_key) paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi179.front.sepia.ceph.com' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIjAE5rOEWC6ycPnRrxvpmPPnPGyXMh0xcrx4x1t0edqn1PYrHA75nS014GS4fzmHWHMIQ1M1+nXPkST2bwsrpQ=', expected 'AAAAC3NzaC1lZDI1NTE5AAAAIDnVB65LQ2ZrorCmRDXT/imYcIqttYXGdAR7PyxJCGep' 2024-09-17T07:01:56.910 ERROR:teuthology.orchestra.connection:Error authenticating with smithi179.front.sepia.ceph.com: BadHostKeyException('smithi179.front.sepia.ceph.com', PKey(alg=ECDSA, bits=256, fp=SHA256:62YHXZ/hIv7z5T30+I7hwDbCw/enRJjcemfW9DQTOfg), PKey(alg=ED25519, bits=256, fp=SHA256:W+6Y/LgnRby4510vqEHFE/6fAReT8MdM2qNnjG5ecjQ)) (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/connection.py", line 111, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 468, in connect raise BadHostKeyException(hostname, server_key, our_key) paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi179.front.sepia.ceph.com' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIjAE5rOEWC6ycPnRrxvpmPPnPGyXMh0xcrx4x1t0edqn1PYrHA75nS014GS4fzmHWHMIQ1M1+nXPkST2bwsrpQ=', expected 'AAAAC3NzaC1lZDI1NTE5AAAAIDnVB65LQ2ZrorCmRDXT/imYcIqttYXGdAR7PyxJCGep' 2024-09-17T07:02:05.520 DEBUG:teuthology.orchestra.remote:timed out 2024-09-17T07:02:15.931 ERROR:teuthology.orchestra.connection:Error authenticating with smithi179.front.sepia.ceph.com: BadHostKeyException('smithi179.front.sepia.ceph.com', PKey(alg=ECDSA, bits=256, fp=SHA256:62YHXZ/hIv7z5T30+I7hwDbCw/enRJjcemfW9DQTOfg), PKey(alg=ED25519, bits=256, fp=SHA256:W+6Y/LgnRby4510vqEHFE/6fAReT8MdM2qNnjG5ecjQ)) (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/connection.py", line 111, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 468, in connect raise BadHostKeyException(hostname, server_key, our_key) paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi179.front.sepia.ceph.com' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIjAE5rOEWC6ycPnRrxvpmPPnPGyXMh0xcrx4x1t0edqn1PYrHA75nS014GS4fzmHWHMIQ1M1+nXPkST2bwsrpQ=', expected 'AAAAC3NzaC1lZDI1NTE5AAAAIDnVB65LQ2ZrorCmRDXT/imYcIqttYXGdAR7PyxJCGep' 2024-09-17T07:02:17.521 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-09-17T07:02:17.522 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T07:02:17.794 DEBUG:teuthology.orchestra.run.smithi044:> true 2024-09-17T07:02:18.143 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-09-17T07:02:18.143 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-17T07:02:18.143 DEBUG:teuthology.orchestra.run.smithi044:> uname -r 2024-09-17T07:02:18.160 INFO:teuthology.orchestra.run.smithi044.stdout:5.14.0-508.el9.x86_64 2024-09-17T07:02:18.160 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-17T07:02:18.160 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T07:02:18.160 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-17T07:02:19.161 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-17T07:02:19.161 DEBUG:teuthology.orchestra.run.smithi044:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T07:02:19.242 INFO:teuthology.orchestra.run.smithi044.stdout:ttyS1 2024-09-17T07:02:19.265 DEBUG:teuthology.parallel:result is None 2024-09-17T07:02:37.951 ERROR:teuthology.orchestra.connection:Error authenticating with smithi179.front.sepia.ceph.com: BadHostKeyException('smithi179.front.sepia.ceph.com', PKey(alg=ECDSA, bits=256, fp=SHA256:62YHXZ/hIv7z5T30+I7hwDbCw/enRJjcemfW9DQTOfg), PKey(alg=ED25519, bits=256, fp=SHA256:W+6Y/LgnRby4510vqEHFE/6fAReT8MdM2qNnjG5ecjQ)) (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/connection.py", line 111, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 468, in connect raise BadHostKeyException(hostname, server_key, our_key) paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi179.front.sepia.ceph.com' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIjAE5rOEWC6ycPnRrxvpmPPnPGyXMh0xcrx4x1t0edqn1PYrHA75nS014GS4fzmHWHMIQ1M1+nXPkST2bwsrpQ=', expected 'AAAAC3NzaC1lZDI1NTE5AAAAIDnVB65LQ2ZrorCmRDXT/imYcIqttYXGdAR7PyxJCGep' 2024-09-17T07:03:02.971 ERROR:teuthology.orchestra.connection:Error authenticating with smithi179.front.sepia.ceph.com: BadHostKeyException('smithi179.front.sepia.ceph.com', PKey(alg=ECDSA, bits=256, fp=SHA256:62YHXZ/hIv7z5T30+I7hwDbCw/enRJjcemfW9DQTOfg), PKey(alg=ED25519, bits=256, fp=SHA256:W+6Y/LgnRby4510vqEHFE/6fAReT8MdM2qNnjG5ecjQ)) (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/connection.py", line 111, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 468, in connect raise BadHostKeyException(hostname, server_key, our_key) paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi179.front.sepia.ceph.com' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIjAE5rOEWC6ycPnRrxvpmPPnPGyXMh0xcrx4x1t0edqn1PYrHA75nS014GS4fzmHWHMIQ1M1+nXPkST2bwsrpQ=', expected 'AAAAC3NzaC1lZDI1NTE5AAAAIDnVB65LQ2ZrorCmRDXT/imYcIqttYXGdAR7PyxJCGep' 2024-09-17T07:03:30.992 ERROR:teuthology.orchestra.connection:Error authenticating with smithi179.front.sepia.ceph.com: BadHostKeyException('smithi179.front.sepia.ceph.com', PKey(alg=ECDSA, bits=256, fp=SHA256:62YHXZ/hIv7z5T30+I7hwDbCw/enRJjcemfW9DQTOfg), PKey(alg=ED25519, bits=256, fp=SHA256:W+6Y/LgnRby4510vqEHFE/6fAReT8MdM2qNnjG5ecjQ)) (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/connection.py", line 111, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 468, in connect raise BadHostKeyException(hostname, server_key, our_key) paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi179.front.sepia.ceph.com' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIjAE5rOEWC6ycPnRrxvpmPPnPGyXMh0xcrx4x1t0edqn1PYrHA75nS014GS4fzmHWHMIQ1M1+nXPkST2bwsrpQ=', expected 'AAAAC3NzaC1lZDI1NTE5AAAAIDnVB65LQ2ZrorCmRDXT/imYcIqttYXGdAR7PyxJCGep' 2024-09-17T07:03:30.992 DEBUG:teuthology.orchestra.remote:'connect to smithi179.front.sepia.ceph.com' reached maximum tries (10) after waiting for 145 seconds 2024-09-17T07:03:39.994 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-09-17T07:03:39.995 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T07:03:40.537 DEBUG:teuthology.orchestra.run.smithi179:> true 2024-09-17T07:03:40.899 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-09-17T07:03:40.899 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-508.el9.x86_64"... 2024-09-17T07:03:40.899 DEBUG:teuthology.orchestra.run.smithi179:> uname -r 2024-09-17T07:03:40.915 INFO:teuthology.orchestra.run.smithi179.stdout:5.14.0-508.el9.x86_64 2024-09-17T07:03:40.916 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-17T07:03:40.916 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T07:03:40.916 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-17T07:03:41.917 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-17T07:03:41.917 DEBUG:teuthology.orchestra.run.smithi179:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T07:03:41.999 INFO:teuthology.orchestra.run.smithi179.stdout:ttyS1 2024-09-17T07:03:42.020 DEBUG:teuthology.parallel:result is None 2024-09-17T07:03:42.021 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-17T07:03:42.028 INFO:teuthology.task.internal:Creating test directory... 2024-09-17T07:03:42.028 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T07:03:42.031 DEBUG:teuthology.orchestra.run.smithi179:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T07:03:42.049 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-17T07:03:42.056 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-17T07:03:42.063 INFO:teuthology.task.internal:Creating archive directory... 2024-09-17T07:03:42.063 DEBUG:teuthology.orchestra.run.smithi044:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T07:03:42.091 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T07:03:42.123 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-17T07:03:42.129 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-17T07:03:42.130 DEBUG:teuthology.orchestra.run.smithi044:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T07:03:42.159 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T07:03:42.193 INFO:teuthology.orchestra.run.smithi044.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T07:03:42.194 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T07:03:42.205 INFO:teuthology.orchestra.run.smithi044.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T07:03:42.205 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T07:03:42.207 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-17T07:03:42.214 INFO:teuthology.task.internal:Configuring sudo... 2024-09-17T07:03:42.214 DEBUG:teuthology.orchestra.run.smithi044:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T07:03:42.249 DEBUG:teuthology.orchestra.run.smithi179:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T07:03:42.276 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-17T07:03:42.286 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-17T07:03:42.286 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T07:03:42.317 DEBUG:teuthology.orchestra.run.smithi179:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T07:03:42.333 DEBUG:teuthology.orchestra.run.smithi044:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T07:03:42.417 DEBUG:teuthology.orchestra.run.smithi044:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T07:03:42.465 DEBUG:teuthology.orchestra.run.smithi044:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T07:03:42.548 DEBUG:teuthology.orchestra.run.smithi044:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T07:03:42.591 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-09-17T07:03:42.591 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T07:03:42.657 DEBUG:teuthology.orchestra.run.smithi179:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T07:03:42.703 DEBUG:teuthology.orchestra.run.smithi179:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T07:03:42.750 DEBUG:teuthology.orchestra.run.smithi179:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T07:03:42.834 DEBUG:teuthology.orchestra.run.smithi179:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T07:03:42.874 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-09-17T07:03:42.874 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T07:03:42.940 DEBUG:teuthology.orchestra.run.smithi044:> sudo service rsyslog restart 2024-09-17T07:03:42.943 DEBUG:teuthology.orchestra.run.smithi179:> sudo service rsyslog restart 2024-09-17T07:03:43.001 INFO:teuthology.orchestra.run.smithi044.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T07:03:43.036 INFO:teuthology.orchestra.run.smithi179.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T07:03:43.403 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-17T07:03:43.413 INFO:teuthology.task.internal:Starting timer... 2024-09-17T07:03:43.413 INFO:teuthology.run_tasks:Running task pcp... 2024-09-17T07:03:43.426 INFO:teuthology.run_tasks:Running task selinux... 2024-09-17T07:03:43.446 DEBUG:teuthology.orchestra.run.smithi044:> sudo service auditd rotate 2024-09-17T07:03:43.506 INFO:teuthology.orchestra.run.smithi044.stdout:Rotating logs: 2024-09-17T07:03:43.508 DEBUG:teuthology.orchestra.run.smithi179:> sudo service auditd rotate 2024-09-17T07:03:43.568 INFO:teuthology.orchestra.run.smithi179.stdout:Rotating logs: 2024-09-17T07:03:43.570 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-17T07:03:43.570 DEBUG:teuthology.orchestra.run.smithi044:> /usr/sbin/getenforce 2024-09-17T07:03:43.593 INFO:teuthology.orchestra.run.smithi044.stdout:Permissive 2024-09-17T07:03:43.593 DEBUG:teuthology.orchestra.run.smithi179:> /usr/sbin/getenforce 2024-09-17T07:03:43.618 INFO:teuthology.orchestra.run.smithi179.stdout:Permissive 2024-09-17T07:03:43.618 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi044.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi179.front.sepia.ceph.com': 'permissive'} 2024-09-17T07:03:43.619 DEBUG:teuthology.orchestra.run.smithi044:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T07:03:43.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:03:43.662 DEBUG:teuthology.orchestra.run.smithi179:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T07:03:43.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:03:43.688 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-17T07:03:43.689 DEBUG:teuthology.orchestra.run.smithi044:> sudo /usr/sbin/setenforce permissive 2024-09-17T07:03:43.732 DEBUG:teuthology.orchestra.run.smithi179:> sudo /usr/sbin/setenforce permissive 2024-09-17T07:03:43.767 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-17T07:03:43.781 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-17T07:03:43.783 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-17T07:03:43.918 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-17T07:03:43.930 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-17T07:03:43.931 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi044.front.sepia.ceph.com,smithi179.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-17T07:07:30.535 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi044.front.sepia.ceph.com'), Remote(name='ubuntu@smithi179.front.sepia.ceph.com')] 2024-09-17T07:07:30.536 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-09-17T07:07:30.538 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T07:07:30.618 DEBUG:teuthology.orchestra.run.smithi044:> true 2024-09-17T07:07:30.697 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-09-17T07:07:30.698 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-09-17T07:07:30.699 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T07:07:30.773 DEBUG:teuthology.orchestra.run.smithi179:> true 2024-09-17T07:07:30.852 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-09-17T07:07:30.852 INFO:teuthology.run_tasks:Running task clock... 2024-09-17T07:07:30.864 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-17T07:07:30.864 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T07:07:30.865 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T07:07:30.868 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T07:07:30.868 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T07:07:30.897 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T07:07:30.913 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T07:07:30.921 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T07:07:30.936 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T07:07:30.940 INFO:teuthology.orchestra.run.smithi044.stderr:sudo: ntpd: command not found 2024-09-17T07:07:30.951 INFO:teuthology.orchestra.run.smithi044.stdout:506 Cannot talk to daemon 2024-09-17T07:07:30.961 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: ntpd: command not found 2024-09-17T07:07:30.967 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T07:07:30.972 INFO:teuthology.orchestra.run.smithi179.stdout:506 Cannot talk to daemon 2024-09-17T07:07:30.982 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T07:07:30.987 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T07:07:31.001 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T07:07:31.038 INFO:teuthology.orchestra.run.smithi044.stderr:bash: line 1: ntpq: command not found 2024-09-17T07:07:31.042 INFO:teuthology.orchestra.run.smithi044.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T07:07:31.042 INFO:teuthology.orchestra.run.smithi044.stdout:=============================================================================== 2024-09-17T07:07:31.042 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T07:07:31.043 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T07:07:31.043 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T07:07:31.043 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T07:07:31.057 INFO:teuthology.orchestra.run.smithi179.stderr:bash: line 1: ntpq: command not found 2024-09-17T07:07:31.062 INFO:teuthology.orchestra.run.smithi179.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T07:07:31.062 INFO:teuthology.orchestra.run.smithi179.stdout:=============================================================================== 2024-09-17T07:07:31.062 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T07:07:31.062 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T07:07:31.062 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T07:07:31.063 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T07:07:31.063 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-17T07:07:31.152 INFO:tasks.cephadm:Config: {'allow_ptrace': False, 'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v16.2.0', 'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': True}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_STRAY_DAEMON', 'CEPHADM_FAILED_DAEMON', 'CEPHADM_AGENT_DOWN'], 'log-only-match': ['CEPHADM_'], 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9'} 2024-09-17T07:07:31.153 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v16.2.0 2024-09-17T07:07:31.153 INFO:tasks.cephadm:Cluster fsid is 817ae996-74c3-11ef-bceb-c7b262605968 2024-09-17T07:07:31.153 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-17T07:07:31.153 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.44', 'mon.c': '[v2:172.21.15.44:3301,v1:172.21.15.44:6790]', 'mon.b': '172.21.15.179'} 2024-09-17T07:07:31.154 INFO:tasks.cephadm:First mon is mon.a on smithi044 2024-09-17T07:07:31.154 INFO:tasks.cephadm:First mgr is y 2024-09-17T07:07:31.154 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-17T07:07:31.154 DEBUG:teuthology.orchestra.run.smithi044:> sudo hostname $(hostname -s) 2024-09-17T07:07:31.181 DEBUG:teuthology.orchestra.run.smithi179:> sudo hostname $(hostname -s) 2024-09-17T07:07:31.207 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.0)... 2024-09-17T07:07:31.207 DEBUG:teuthology.orchestra.run.smithi044:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-17T07:07:31.447 INFO:teuthology.orchestra.run.smithi044.stdout:-rw-r--r--. 1 ubuntu ubuntu 288386 Sep 17 07:07 /home/ubuntu/cephtest/cephadm 2024-09-17T07:07:31.447 DEBUG:teuthology.orchestra.run.smithi179:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-17T07:07:31.571 INFO:teuthology.orchestra.run.smithi179.stdout:-rw-r--r--. 1 ubuntu ubuntu 288386 Sep 17 07:07 /home/ubuntu/cephtest/cephadm 2024-09-17T07:07:31.571 DEBUG:teuthology.orchestra.run.smithi044:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-17T07:07:31.588 DEBUG:teuthology.orchestra.run.smithi179:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-17T07:07:31.615 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v16.2.0 on all hosts... 2024-09-17T07:07:31.615 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 pull 2024-09-17T07:07:31.631 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 pull 2024-09-17T07:07:32.471 INFO:teuthology.orchestra.run.smithi044.stderr:Pulling container image quay.io/ceph/ceph:v16.2.0... 2024-09-17T07:07:32.533 INFO:teuthology.orchestra.run.smithi179.stderr:Pulling container image quay.io/ceph/ceph:v16.2.0... 2024-09-17T07:07:44.442 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-09-17T07:07:44.443 INFO:teuthology.orchestra.run.smithi179.stdout: "ceph_version": "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)", 2024-09-17T07:07:44.443 INFO:teuthology.orchestra.run.smithi179.stdout: "image_id": "24ecd6d5f14c2268c5d2c666aa151456949cf76c6747d873c6fcf41f6f672d15", 2024-09-17T07:07:44.443 INFO:teuthology.orchestra.run.smithi179.stdout: "repo_digests": [ 2024-09-17T07:07:44.443 INFO:teuthology.orchestra.run.smithi179.stdout: "quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000", 2024-09-17T07:07:44.443 INFO:teuthology.orchestra.run.smithi179.stdout: "quay.io/ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a" 2024-09-17T07:07:44.443 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-09-17T07:07:44.443 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-09-17T07:07:44.869 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:07:44.870 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_version": "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)", 2024-09-17T07:07:44.870 INFO:teuthology.orchestra.run.smithi044.stdout: "image_id": "24ecd6d5f14c2268c5d2c666aa151456949cf76c6747d873c6fcf41f6f672d15", 2024-09-17T07:07:44.870 INFO:teuthology.orchestra.run.smithi044.stdout: "repo_digests": [ 2024-09-17T07:07:44.870 INFO:teuthology.orchestra.run.smithi044.stdout: "quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000", 2024-09-17T07:07:44.870 INFO:teuthology.orchestra.run.smithi044.stdout: "quay.io/ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a" 2024-09-17T07:07:44.870 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-09-17T07:07:44.870 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:07:44.888 DEBUG:teuthology.orchestra.run.smithi044:> sudo mkdir -p /etc/ceph 2024-09-17T07:07:44.938 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /etc/ceph 2024-09-17T07:07:44.971 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 777 /etc/ceph 2024-09-17T07:07:44.997 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 777 /etc/ceph 2024-09-17T07:07:45.039 INFO:tasks.cephadm:Writing seed config... 2024-09-17T07:07:45.040 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-09-17T07:07:45.040 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-17T07:07:45.040 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-17T07:07:45.040 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = True 2024-09-17T07:07:45.041 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-17T07:07:45.041 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-17T07:07:45.041 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-17T07:07:45.041 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-17T07:07:45.041 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-17T07:07:45.041 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-09-17T07:07:45.042 DEBUG:teuthology.orchestra.run.smithi044:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-17T07:07:45.057 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 817ae996-74c3-11ef-bceb-c7b262605968 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = True [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-09-17T07:07:45.058 DEBUG:teuthology.orchestra.run.smithi044:mon.a> sudo journalctl -f -n 0 -u ceph-817ae996-74c3-11ef-bceb-c7b262605968@mon.a.service 2024-09-17T07:07:45.100 DEBUG:teuthology.orchestra.run.smithi044:mgr.y> sudo journalctl -f -n 0 -u ceph-817ae996-74c3-11ef-bceb-c7b262605968@mgr.y.service 2024-09-17T07:07:45.143 INFO:tasks.cephadm:Bootstrapping... 2024-09-17T07:07:45.143 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 -v bootstrap --fsid 817ae996-74c3-11ef-bceb-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.44 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-17T07:07:45.333 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /bin/podman --version 2024-09-17T07:07:45.354 INFO:teuthology.orchestra.run.smithi044.stderr:/bin/podman: podman version 5.2.2 2024-09-17T07:07:45.355 INFO:teuthology.orchestra.run.smithi044.stderr:Verifying podman|docker is present... 2024-09-17T07:07:45.355 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /bin/podman --version 2024-09-17T07:07:45.373 INFO:teuthology.orchestra.run.smithi044.stderr:/bin/podman: podman version 5.2.2 2024-09-17T07:07:45.375 INFO:teuthology.orchestra.run.smithi044.stderr:Verifying lvm2 is present... 2024-09-17T07:07:45.375 INFO:teuthology.orchestra.run.smithi044.stderr:Verifying time synchronization is in place... 2024-09-17T07:07:45.375 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: systemctl is-enabled chrony.service 2024-09-17T07:07:45.382 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-09-17T07:07:45.383 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: systemctl is-active chrony.service 2024-09-17T07:07:45.389 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: inactive 2024-09-17T07:07:45.389 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: systemctl is-enabled chronyd.service 2024-09-17T07:07:45.396 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: enabled 2024-09-17T07:07:45.397 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: systemctl is-active chronyd.service 2024-09-17T07:07:45.403 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: active 2024-09-17T07:07:45.403 INFO:teuthology.orchestra.run.smithi044.stderr:Unit chronyd.service is enabled and running 2024-09-17T07:07:45.403 INFO:teuthology.orchestra.run.smithi044.stderr:Repeating the final host check... 2024-09-17T07:07:45.403 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /bin/podman --version 2024-09-17T07:07:45.422 INFO:teuthology.orchestra.run.smithi044.stderr:/bin/podman: podman version 5.2.2 2024-09-17T07:07:45.423 INFO:teuthology.orchestra.run.smithi044.stderr:podman|docker (/bin/podman) is present 2024-09-17T07:07:45.423 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl is present 2024-09-17T07:07:45.423 INFO:teuthology.orchestra.run.smithi044.stderr:lvcreate is present 2024-09-17T07:07:45.424 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: systemctl is-enabled chrony.service 2024-09-17T07:07:45.429 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-09-17T07:07:45.430 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: systemctl is-active chrony.service 2024-09-17T07:07:45.436 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: inactive 2024-09-17T07:07:45.436 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: systemctl is-enabled chronyd.service 2024-09-17T07:07:45.443 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: enabled 2024-09-17T07:07:45.444 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: systemctl is-active chronyd.service 2024-09-17T07:07:45.450 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: active 2024-09-17T07:07:45.451 INFO:teuthology.orchestra.run.smithi044.stderr:Unit chronyd.service is enabled and running 2024-09-17T07:07:45.451 INFO:teuthology.orchestra.run.smithi044.stderr:Host looks OK 2024-09-17T07:07:45.451 INFO:teuthology.orchestra.run.smithi044.stderr:Cluster fsid: 817ae996-74c3-11ef-bceb-c7b262605968 2024-09-17T07:07:45.451 INFO:teuthology.orchestra.run.smithi044.stderr:Acquiring lock 140421386999840 on /run/cephadm/817ae996-74c3-11ef-bceb-c7b262605968.lock 2024-09-17T07:07:45.451 INFO:teuthology.orchestra.run.smithi044.stderr:Lock 140421386999840 acquired on /run/cephadm/817ae996-74c3-11ef-bceb-c7b262605968.lock 2024-09-17T07:07:45.451 INFO:teuthology.orchestra.run.smithi044.stderr:Verifying IP 172.21.15.44 port 3300 ... 2024-09-17T07:07:45.451 INFO:teuthology.orchestra.run.smithi044.stderr:Verifying IP 172.21.15.44 port 6789 ... 2024-09-17T07:07:45.452 INFO:teuthology.orchestra.run.smithi044.stderr:Base mon IP is 172.21.15.44, final addrv is [v2:172.21.15.44:3300,v1:172.21.15.44:6789] 2024-09-17T07:07:45.452 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /sbin/ip route ls 2024-09-17T07:07:45.454 INFO:teuthology.orchestra.run.smithi044.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.44 metric 100 2024-09-17T07:07:45.454 INFO:teuthology.orchestra.run.smithi044.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.44 metric 100 2024-09-17T07:07:45.455 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /sbin/ip -6 route ls 2024-09-17T07:07:45.457 INFO:teuthology.orchestra.run.smithi044.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-09-17T07:07:45.457 INFO:teuthology.orchestra.run.smithi044.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-09-17T07:07:45.457 INFO:teuthology.orchestra.run.smithi044.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1792sec hoplimit 64 pref medium 2024-09-17T07:07:45.457 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /sbin/ip -6 addr ls 2024-09-17T07:07:45.460 INFO:teuthology.orchestra.run.smithi044.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-17T07:07:45.460 INFO:teuthology.orchestra.run.smithi044.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-09-17T07:07:45.460 INFO:teuthology.orchestra.run.smithi044.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-09-17T07:07:45.460 INFO:teuthology.orchestra.run.smithi044.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2024-09-17T07:07:45.460 INFO:teuthology.orchestra.run.smithi044.stderr:/sbin/ip: inet6 fe80::ec4:7aff:febd:1572/64 scope link 2024-09-17T07:07:45.460 INFO:teuthology.orchestra.run.smithi044.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-09-17T07:07:45.461 INFO:teuthology.orchestra.run.smithi044.stderr:Mon IP 172.21.15.44 is in CIDR network 172.21.0.0/20 2024-09-17T07:07:45.461 INFO:teuthology.orchestra.run.smithi044.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-17T07:07:45.462 INFO:teuthology.orchestra.run.smithi044.stderr:Pulling container image quay.io/ceph/ceph:v16.2.0... 2024-09-17T07:07:45.462 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /bin/podman pull quay.io/ceph/ceph:v16.2.0 2024-09-17T07:07:45.565 INFO:teuthology.orchestra.run.smithi044.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v16.2.0... 2024-09-17T07:07:45.880 INFO:teuthology.orchestra.run.smithi044.stderr:/bin/podman: Getting image source signatures 2024-09-17T07:07:45.880 INFO:teuthology.orchestra.run.smithi044.stderr:/bin/podman: Copying blob sha256:958e3c20ad47ec7f3ec0446a4a630f2285dd6b0a4c969f4f702f7c2e78f3d68b 2024-09-17T07:07:45.880 INFO:teuthology.orchestra.run.smithi044.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2024-09-17T07:07:45.880 INFO:teuthology.orchestra.run.smithi044.stderr:/bin/podman: Copying config sha256:24ecd6d5f14c2268c5d2c666aa151456949cf76c6747d873c6fcf41f6f672d15 2024-09-17T07:07:46.040 INFO:teuthology.orchestra.run.smithi044.stderr:/bin/podman: Writing manifest to image destination 2024-09-17T07:07:46.194 INFO:teuthology.orchestra.run.smithi044.stderr:/bin/podman: 24ecd6d5f14c2268c5d2c666aa151456949cf76c6747d873c6fcf41f6f672d15 2024-09-17T07:07:46.198 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi044 -e CEPH_USE_RANDOM_NONCE=1 quay.io/ceph/ceph:v16.2.0 --version 2024-09-17T07:07:46.569 INFO:teuthology.orchestra.run.smithi044.stderr:ceph: ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable) 2024-09-17T07:07:47.096 INFO:teuthology.orchestra.run.smithi044.stderr:Ceph version: ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable) 2024-09-17T07:07:47.096 INFO:teuthology.orchestra.run.smithi044.stderr:Extracting ceph user uid/gid from container image... 2024-09-17T07:07:47.096 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi044 -e CEPH_USE_RANDOM_NONCE=1 quay.io/ceph/ceph:v16.2.0 -c %u %g /var/lib/ceph 2024-09-17T07:07:47.493 INFO:teuthology.orchestra.run.smithi044.stderr:stat: 167 167 2024-09-17T07:07:48.083 INFO:teuthology.orchestra.run.smithi044.stderr:Creating initial keys... 2024-09-17T07:07:48.083 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi044 -e CEPH_USE_RANDOM_NONCE=1 quay.io/ceph/ceph:v16.2.0 --gen-print-key 2024-09-17T07:07:48.440 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-authtool: AQDEKulmfq9KFRAA85UzH7vJf83FtmIWgWehtw== 2024-09-17T07:07:49.091 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi044 -e CEPH_USE_RANDOM_NONCE=1 quay.io/ceph/ceph:v16.2.0 --gen-print-key 2024-09-17T07:07:53.056 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-authtool: AQDIKulmvAx1BBAA0k9W0/a76KpKQ74WFqYVVw== 2024-09-17T07:07:58.834 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi044 -e CEPH_USE_RANDOM_NONCE=1 quay.io/ceph/ceph:v16.2.0 --gen-print-key 2024-09-17T07:08:00.587 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-authtool: AQDQKulmZ2QKHhAAt/5+ypH6DJyl9ZwL9DeyUA== 2024-09-17T07:08:01.036 INFO:teuthology.orchestra.run.smithi044.stderr:Creating initial monmap... 2024-09-17T07:08:01.037 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi044 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmpymz4ahc6:/tmp/monmap:z quay.io/ceph/ceph:v16.2.0 --create --clobber --fsid 817ae996-74c3-11ef-bceb-c7b262605968 --addv a [v2:172.21.15.44:3300,v1:172.21.15.44:6789] /tmp/monmap 2024-09-17T07:08:01.467 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-17T07:08:01.467 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 817ae996-74c3-11ef-bceb-c7b262605968 2024-09-17T07:08:01.468 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-17T07:08:01.982 INFO:teuthology.orchestra.run.smithi044.stderr:monmaptool for a [v2:172.21.15.44:3300,v1:172.21.15.44:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-17T07:08:01.982 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/monmaptool: set fsid to 817ae996-74c3-11ef-bceb-c7b262605968 2024-09-17T07:08:01.982 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-17T07:08:01.982 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-09-17T07:08:01.983 INFO:teuthology.orchestra.run.smithi044.stderr:Creating mon... 2024-09-17T07:08:01.985 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi044 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/817ae996-74c3-11ef-bceb-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/817ae996-74c3-11ef-bceb-c7b262605968/mon.a:/var/lib/ceph/mon/ceph-a:z -v /tmp/ceph-tmpu2zmnb1w:/tmp/keyring:z -v /tmp/ceph-tmpymz4ahc6:/tmp/monmap:z quay.io/ceph/ceph:v16.2.0 --mkfs -i a --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2024-09-17T07:08:02.390 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.330+0000 7f301b7ba700 0 set uid:gid to 167:167 (ceph:ceph) 2024-09-17T07:08:02.390 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.330+0000 7f301b7ba700 1 imported monmap: 2024-09-17T07:08:02.390 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: epoch 0 2024-09-17T07:08:02.390 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: fsid 817ae996-74c3-11ef-bceb-c7b262605968 2024-09-17T07:08:02.390 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: last_changed 2024-09-17T07:08:01.380178+0000 2024-09-17T07:08:02.391 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: created 2024-09-17T07:08:01.380178+0000 2024-09-17T07:08:02.416 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-09-17T07:08:02.416 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-09-17T07:08:02.416 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.a 2024-09-17T07:08:02.416 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: 2024-09-17T07:08:02.416 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.330+0000 7f301b7ba700 0 /usr/bin/ceph-mon: set fsid to 817ae996-74c3-11ef-bceb-c7b262605968 2024-09-17T07:08:02.417 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: RocksDB version: 6.8.1 2024-09-17T07:08:02.417 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: 2024-09-17T07:08:02.417 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-17T07:08:02.417 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Compile date Mar 30 2021 2024-09-17T07:08:02.417 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: DB SUMMARY 2024-09-17T07:08:02.417 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: 2024-09-17T07:08:02.417 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 0, files: 2024-09-17T07:08:02.417 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: 2024-09-17T07:08:02.417 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 2024-09-17T07:08:02.417 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: 2024-09-17T07:08:02.417 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.error_if_exists: 0 2024-09-17T07:08:02.417 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.create_if_missing: 1 2024-09-17T07:08:02.417 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.paranoid_checks: 1 2024-09-17T07:08:02.418 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.env: 0x55c40139d1c0 2024-09-17T07:08:02.418 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.fs: Posix File System 2024-09-17T07:08:02.418 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.info_log: 0x55c403992b80 2024-09-17T07:08:02.418 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.max_file_opening_threads: 16 2024-09-17T07:08:02.418 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.statistics: (nil) 2024-09-17T07:08:02.418 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.use_fsync: 0 2024-09-17T07:08:02.418 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.max_log_file_size: 0 2024-09-17T07:08:02.418 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-17T07:08:02.418 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-09-17T07:08:02.418 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.keep_log_file_num: 1000 2024-09-17T07:08:02.418 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.recycle_log_file_num: 0 2024-09-17T07:08:02.418 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.allow_fallocate: 1 2024-09-17T07:08:02.418 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.allow_mmap_reads: 0 2024-09-17T07:08:02.418 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.allow_mmap_writes: 0 2024-09-17T07:08:02.419 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.use_direct_reads: 0 2024-09-17T07:08:02.419 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-17T07:08:02.419 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.create_missing_column_families: 0 2024-09-17T07:08:02.419 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.db_log_dir: 2024-09-17T07:08:02.419 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-09-17T07:08:02.419 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-09-17T07:08:02.419 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.max_subcompactions: 1 2024-09-17T07:08:02.419 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.max_background_flushes: -1 2024-09-17T07:08:02.419 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-09-17T07:08:02.419 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-09-17T07:08:02.419 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-17T07:08:02.419 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-17T07:08:02.419 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-09-17T07:08:02.419 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.advise_random_on_open: 1 2024-09-17T07:08:02.419 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.db_write_buffer_size: 0 2024-09-17T07:08:02.420 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.write_buffer_manager: 0x55c40399d410 2024-09-17T07:08:02.420 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-17T07:08:02.420 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-17T07:08:02.420 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-17T07:08:02.420 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-09-17T07:08:02.420 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.rate_limiter: (nil) 2024-09-17T07:08:02.420 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-17T07:08:02.420 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.wal_recovery_mode: 2 2024-09-17T07:08:02.420 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.enable_thread_tracking: 0 2024-09-17T07:08:02.420 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.enable_pipelined_write: 0 2024-09-17T07:08:02.420 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.unordered_write: 0 2024-09-17T07:08:02.420 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-17T07:08:02.420 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-17T07:08:02.421 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-17T07:08:02.421 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-17T07:08:02.421 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.row_cache: None 2024-09-17T07:08:02.421 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.wal_filter: None 2024-09-17T07:08:02.421 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-17T07:08:02.421 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.allow_ingest_behind: 0 2024-09-17T07:08:02.421 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.preserve_deletes: 0 2024-09-17T07:08:02.421 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.two_write_queues: 0 2024-09-17T07:08:02.421 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.manual_wal_flush: 0 2024-09-17T07:08:02.421 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.atomic_flush: 0 2024-09-17T07:08:02.421 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-17T07:08:02.421 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-09-17T07:08:02.421 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-09-17T07:08:02.421 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.log_readahead_size: 0 2024-09-17T07:08:02.422 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-09-17T07:08:02.422 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.max_background_jobs: 2 2024-09-17T07:08:02.422 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.max_background_compactions: -1 2024-09-17T07:08:02.422 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-17T07:08:02.422 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-17T07:08:02.422 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-09-17T07:08:02.422 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.max_total_wal_size: 0 2024-09-17T07:08:02.422 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-17T07:08:02.422 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-09-17T07:08:02.422 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-09-17T07:08:02.422 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-17T07:08:02.422 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.max_open_files: -1 2024-09-17T07:08:02.422 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.bytes_per_sync: 0 2024-09-17T07:08:02.423 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-09-17T07:08:02.423 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-09-17T07:08:02.423 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Options.compaction_readahead_size: 0 2024-09-17T07:08:02.423 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Compression algorithms supported: 2024-09-17T07:08:02.423 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-17T07:08:02.423 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: kZSTD supported: 0 2024-09-17T07:08:02.423 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: kXpressCompression supported: 0 2024-09-17T07:08:02.423 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: kLZ4HCCompression supported: 1 2024-09-17T07:08:02.423 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: kLZ4Compression supported: 1 2024-09-17T07:08:02.423 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: kBZip2Compression supported: 0 2024-09-17T07:08:02.423 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: kZlibCompression supported: 1 2024-09-17T07:08:02.423 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: kSnappyCompression supported: 1 2024-09-17T07:08:02.423 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.331+0000 7f301b7ba700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-09-17T07:08:02.423 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-09-17T07:08:02.423 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: 2024-09-17T07:08:02.424 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 2024-09-17T07:08:02.424 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: 2024-09-17T07:08:02.424 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-09-17T07:08:02.424 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: 2024-09-17T07:08:02.424 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-17T07:08:02.424 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.merge_operator: 2024-09-17T07:08:02.424 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.compaction_filter: None 2024-09-17T07:08:02.424 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.compaction_filter_factory: None 2024-09-17T07:08:02.424 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-09-17T07:08:02.424 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-09-17T07:08:02.424 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c4038ac0d0) 2024-09-17T07:08:02.424 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-09-17T07:08:02.425 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-17T07:08:02.425 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-17T07:08:02.425 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-09-17T07:08:02.425 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: index_type: 0 2024-09-17T07:08:02.425 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-09-17T07:08:02.425 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-09-17T07:08:02.425 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-09-17T07:08:02.425 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-09-17T07:08:02.425 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: checksum: 1 2024-09-17T07:08:02.425 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-09-17T07:08:02.425 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: block_cache: 0x55c4038e2d10 2024-09-17T07:08:02.425 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-09-17T07:08:02.425 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-09-17T07:08:02.425 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-09-17T07:08:02.426 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-09-17T07:08:02.426 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-09-17T07:08:02.426 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-09-17T07:08:02.426 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-09-17T07:08:02.426 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-09-17T07:08:02.426 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-09-17T07:08:02.426 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-09-17T07:08:02.426 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-09-17T07:08:02.426 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-09-17T07:08:02.426 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-09-17T07:08:02.426 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-09-17T07:08:02.426 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-09-17T07:08:02.426 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-17T07:08:02.427 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-09-17T07:08:02.427 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-09-17T07:08:02.427 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-09-17T07:08:02.427 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: format_version: 2 2024-09-17T07:08:02.427 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-09-17T07:08:02.427 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: block_align: 0 2024-09-17T07:08:02.427 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: 2024-09-17T07:08:02.427 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.write_buffer_size: 33554432 2024-09-17T07:08:02.427 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.max_write_buffer_number: 2 2024-09-17T07:08:02.427 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.compression: NoCompression 2024-09-17T07:08:02.427 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.bottommost_compression: Disabled 2024-09-17T07:08:02.427 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.prefix_extractor: nullptr 2024-09-17T07:08:02.427 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-17T07:08:02.428 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.num_levels: 7 2024-09-17T07:08:02.428 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-17T07:08:02.428 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-17T07:08:02.428 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-17T07:08:02.428 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-17T07:08:02.428 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-17T07:08:02.428 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-17T07:08:02.428 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-17T07:08:02.428 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-17T07:08:02.428 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-17T07:08:02.428 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-09-17T07:08:02.428 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.compression_opts.level: 32767 2024-09-17T07:08:02.429 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.compression_opts.strategy: 0 2024-09-17T07:08:02.429 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-17T07:08:02.429 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-17T07:08:02.429 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.compression_opts.enabled: false 2024-09-17T07:08:02.429 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-17T07:08:02.429 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-17T07:08:02.429 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-17T07:08:02.429 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.target_file_size_base: 67108864 2024-09-17T07:08:02.429 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-09-17T07:08:02.429 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-17T07:08:02.429 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-17T07:08:02.429 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-17T07:08:02.429 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-17T07:08:02.430 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-17T07:08:02.430 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-17T07:08:02.430 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-17T07:08:02.430 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-17T07:08:02.430 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-17T07:08:02.430 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-17T07:08:02.430 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-17T07:08:02.430 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-17T07:08:02.430 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.arena_block_size: 4194304 2024-09-17T07:08:02.430 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-17T07:08:02.430 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-17T07:08:02.430 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-17T07:08:02.430 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.disable_auto_compactions: 0 2024-09-17T07:08:02.430 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-17T07:08:02.431 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-17T07:08:02.431 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-17T07:08:02.431 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-17T07:08:02.431 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-17T07:08:02.431 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-17T07:08:02.431 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-17T07:08:02.431 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-17T07:08:02.431 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-17T07:08:02.431 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-17T07:08:02.431 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.table_properties_collectors: 2024-09-17T07:08:02.431 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.inplace_update_support: 0 2024-09-17T07:08:02.431 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-09-17T07:08:02.431 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-17T07:08:02.432 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-17T07:08:02.432 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-09-17T07:08:02.432 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.bloom_locality: 0 2024-09-17T07:08:02.432 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.max_successive_merges: 0 2024-09-17T07:08:02.432 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-09-17T07:08:02.432 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.paranoid_file_checks: 0 2024-09-17T07:08:02.432 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.force_consistency_checks: 0 2024-09-17T07:08:02.432 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.report_bg_io_stats: 0 2024-09-17T07:08:02.432 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.ttl: 2592000 2024-09-17T07:08:02.432 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.332+0000 7f301b7ba700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-09-17T07:08:02.432 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.333+0000 7f301b7ba700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-17T07:08:02.432 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: 2024-09-17T07:08:02.433 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.333+0000 7f301b7ba700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-09-17T07:08:02.433 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: 2024-09-17T07:08:02.433 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.333+0000 7f301b7ba700 4 rocksdb: DB pointer 0x55c4039a7800 2024-09-17T07:08:02.433 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.334+0000 7f3004655700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-09-17T07:08:02.433 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.334+0000 7f3004655700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-09-17T07:08:02.433 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-09-17T07:08:02.433 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T07:08:02.433 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-17T07:08:02.433 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T07:08:02.433 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T07:08:02.434 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-17T07:08:02.434 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-17T07:08:02.434 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T07:08:02.434 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: 2024-09-17T07:08:02.434 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-09-17T07:08:02.434 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-17T07:08:02.434 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T07:08:02.434 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-17T07:08:02.434 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-17T07:08:02.434 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: 2024-09-17T07:08:02.434 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-09-17T07:08:02.434 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-17T07:08:02.434 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T07:08:02.434 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T07:08:02.435 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-09-17T07:08:02.435 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T07:08:02.435 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T07:08:02.435 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T07:08:02.435 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-09-17T07:08:02.435 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T07:08:02.435 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T07:08:02.435 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T07:08:02.435 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: 2024-09-17T07:08:02.435 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-09-17T07:08:02.435 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: 2024-09-17T07:08:02.435 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-09-17T07:08:02.435 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-17T07:08:02.435 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T07:08:02.436 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-17T07:08:02.436 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-17T07:08:02.436 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: 2024-09-17T07:08:02.436 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-09-17T07:08:02.436 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-17T07:08:02.436 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T07:08:02.436 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T07:08:02.436 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-09-17T07:08:02.436 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T07:08:02.436 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T07:08:02.436 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T07:08:02.436 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-09-17T07:08:02.436 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T07:08:02.436 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T07:08:02.437 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T07:08:02.437 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: 2024-09-17T07:08:02.437 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-09-17T07:08:02.437 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: 2024-09-17T07:08:02.437 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.335+0000 7f301b7ba700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-09-17T07:08:02.437 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.335+0000 7f301b7ba700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-09-17T07:08:02.437 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph-mon: debug 2024-09-17T07:08:02.335+0000 7f301b7ba700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-a for mon.a 2024-09-17T07:08:03.132 INFO:teuthology.orchestra.run.smithi044.stderr:create mon.a on 2024-09-17T07:08:03.133 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/817ae996-74c3-11ef-bceb-c7b262605968 2024-09-17T07:08:03.139 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: systemctl disable ceph.target 2024-09-17T07:08:03.333 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: systemctl enable ceph.target 2024-09-17T07:08:03.341 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-17T07:08:03.477 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: systemctl start ceph.target 2024-09-17T07:08:03.485 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: systemctl enable ceph-817ae996-74c3-11ef-bceb-c7b262605968.target 2024-09-17T07:08:03.493 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-817ae996-74c3-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-817ae996-74c3-11ef-bceb-c7b262605968.target. 2024-09-17T07:08:03.493 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-817ae996-74c3-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-817ae996-74c3-11ef-bceb-c7b262605968.target. 2024-09-17T07:08:03.629 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: systemctl start ceph-817ae996-74c3-11ef-bceb-c7b262605968.target 2024-09-17T07:08:03.654 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: systemctl daemon-reload 2024-09-17T07:08:03.805 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: systemctl stop ceph-817ae996-74c3-11ef-bceb-c7b262605968@mon.a 2024-09-17T07:08:03.820 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: systemctl reset-failed ceph-817ae996-74c3-11ef-bceb-c7b262605968@mon.a 2024-09-17T07:08:03.826 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: Failed to reset failed state of unit ceph-817ae996-74c3-11ef-bceb-c7b262605968@mon.a.service: Unit ceph-817ae996-74c3-11ef-bceb-c7b262605968@mon.a.service not loaded. 2024-09-17T07:08:03.826 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: systemctl enable ceph-817ae996-74c3-11ef-bceb-c7b262605968@mon.a 2024-09-17T07:08:03.833 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: Created symlink /etc/systemd/system/ceph-817ae996-74c3-11ef-bceb-c7b262605968.target.wants/ceph-817ae996-74c3-11ef-bceb-c7b262605968@mon.a.service → /etc/systemd/system/ceph-817ae996-74c3-11ef-bceb-c7b262605968@.service. 2024-09-17T07:08:03.977 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: systemctl start ceph-817ae996-74c3-11ef-bceb-c7b262605968@mon.a 2024-09-17T07:08:04.419 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:04 smithi044 podman[23309]: 2024-09-17 07:08:04.369718433 +0000 UTC m=+0.316122643 container start 3b36ced0728adfb2222457184c9bb2fde6f49b7bfc48a50017882cb777bfa86a (image=quay.io/ceph/ceph:v16.2.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, GIT_BRANCH=HEAD, org.label-schema.build-date=20201204, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Base Image, RELEASE=HEAD, CEPH_POINT_RELEASE=-16.2.0, org.label-schema.schema-version=1.0, maintainer=Dimitri Savineau ) 2024-09-17T07:08:04.419 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:04 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23323]: cluster 2024-09-17T07:08:04.408272+0000 mon.a (mon.0) 0 : [INF] mkfs 817ae996-74c3-11ef-bceb-c7b262605968 2024-09-17T07:08:04.471 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: systemctl is-enabled firewalld.service 2024-09-17T07:08:04.481 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: disabled 2024-09-17T07:08:04.481 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: systemctl is-active firewalld.service 2024-09-17T07:08:04.489 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: inactive 2024-09-17T07:08:04.490 INFO:teuthology.orchestra.run.smithi044.stderr:firewalld.service is not enabled 2024-09-17T07:08:04.490 INFO:teuthology.orchestra.run.smithi044.stderr:Not possible to enable service . firewalld.service is not available 2024-09-17T07:08:04.490 INFO:teuthology.orchestra.run.smithi044.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-09-17T07:08:04.491 INFO:teuthology.orchestra.run.smithi044.stderr:Waiting for mon to start... 2024-09-17T07:08:04.491 INFO:teuthology.orchestra.run.smithi044.stderr:Waiting for mon... 2024-09-17T07:08:04.491 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi044 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/817ae996-74c3-11ef-bceb-c7b262605968/mon.a:/var/lib/ceph/mon/ceph-a:z -v /tmp/ceph-tmpboltjppk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpukm_xx4s:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 status 2024-09-17T07:08:04.721 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:04 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23323]: cluster 2024-09-17T07:08:04.402662+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-09-17T07:08:04.721 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:04 smithi044 bash[23309]: 3b36ced0728adfb2222457184c9bb2fde6f49b7bfc48a50017882cb777bfa86a 2024-09-17T07:08:04.721 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:04 smithi044 systemd[1]: Started Ceph mon.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:08:05.032 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: cluster: 2024-09-17T07:08:05.032 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: id: 817ae996-74c3-11ef-bceb-c7b262605968 2024-09-17T07:08:05.033 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-09-17T07:08:05.033 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: 2024-09-17T07:08:05.033 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: services: 2024-09-17T07:08:05.033 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.623605s) 2024-09-17T07:08:05.033 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: mgr: no daemons active 2024-09-17T07:08:05.033 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-09-17T07:08:05.033 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: 2024-09-17T07:08:05.033 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: data: 2024-09-17T07:08:05.033 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-09-17T07:08:05.033 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-09-17T07:08:05.033 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-09-17T07:08:05.033 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: pgs: 2024-09-17T07:08:05.033 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: 2024-09-17T07:08:05.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23323]: cluster 2024-09-17T07:08:04.406800+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-09-17T07:08:05.587 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23323]: cluster 2024-09-17T07:08:04.407249+0000 mon.a (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {a=[v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0]} 2024-09-17T07:08:05.587 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23323]: cluster 2024-09-17T07:08:04.410359+0000 mon.a (mon.0) 4 : cluster [DBG] fsmap 2024-09-17T07:08:05.587 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23323]: cluster 2024-09-17T07:08:04.421768+0000 mon.a (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-09-17T07:08:05.587 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23323]: cluster 2024-09-17T07:08:04.423804+0000 mon.a (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-09-17T07:08:05.587 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23323]: audit 2024-09-17T07:08:05.030365+0000 mon.a (mon.0) 7 : audit [DBG] from='client.? 172.21.15.44:0/138174825' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-17T07:08:06.086 INFO:teuthology.orchestra.run.smithi044.stderr:mon is available 2024-09-17T07:08:06.086 INFO:teuthology.orchestra.run.smithi044.stderr:Assimilating anything we can from ceph.conf... 2024-09-17T07:08:06.086 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi044 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/817ae996-74c3-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmpboltjppk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpukm_xx4s:/etc/ceph/ceph.conf:z -v /var/lib/ceph/817ae996-74c3-11ef-bceb-c7b262605968/mon.a:/var/lib/ceph/mon/ceph-a:z quay.io/ceph/ceph:v16.2.0 config assimilate-conf -i /var/lib/ceph/mon/ceph-a/config 2024-09-17T07:08:06.588 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: 2024-09-17T07:08:06.588 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: [global] 2024-09-17T07:08:06.589 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: fsid = 817ae996-74c3-11ef-bceb-c7b262605968 2024-09-17T07:08:06.589 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: log_to_journald = false 2024-09-17T07:08:06.589 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.44:3300,v1:172.21.15.44:6789] 2024-09-17T07:08:06.589 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-09-17T07:08:06.589 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-09-17T07:08:06.589 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-09-17T07:08:06.589 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-09-17T07:08:06.589 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: 2024-09-17T07:08:06.589 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: [mgr] 2024-09-17T07:08:06.589 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = True 2024-09-17T07:08:06.589 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-09-17T07:08:06.589 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: 2024-09-17T07:08:06.589 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: [mon] 2024-09-17T07:08:06.589 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T07:08:06.590 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T07:08:06.590 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: 2024-09-17T07:08:06.590 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: [osd] 2024-09-17T07:08:06.590 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-09-17T07:08:06.590 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2024-09-17T07:08:06.590 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-09-17T07:08:06.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23323]: audit 2024-09-17T07:08:06.583362+0000 mon.a (mon.0) 8 : audit [INF] from='client.? 172.21.15.44:0/146560143' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-17T07:08:06.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23323]: audit 2024-09-17T07:08:06.585795+0000 mon.a (mon.0) 9 : audit [INF] from='client.? 172.21.15.44:0/146560143' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-17T07:08:07.157 INFO:teuthology.orchestra.run.smithi044.stderr:Generating new minimal ceph.conf... 2024-09-17T07:08:07.157 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi044 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/817ae996-74c3-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmpboltjppk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpukm_xx4s:/etc/ceph/ceph.conf:z -v /var/lib/ceph/817ae996-74c3-11ef-bceb-c7b262605968/mon.a:/var/lib/ceph/mon/ceph-a:z quay.io/ceph/ceph:v16.2.0 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-a/config 2024-09-17T07:08:07.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23323]: audit 2024-09-17T07:08:07.665257+0000 mon.a (mon.0) 10 : audit [DBG] from='client.? 172.21.15.44:0/1580534661' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:08:08.213 INFO:teuthology.orchestra.run.smithi044.stderr:Restarting the monitor... 2024-09-17T07:08:08.213 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: systemctl restart ceph-817ae996-74c3-11ef-bceb-c7b262605968@mon.a 2024-09-17T07:08:08.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:08 smithi044 systemd[1]: Stopping Ceph mon.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:08:08.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23323]: debug 2024-09-17T07:08:08.354+0000 7fddc745c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-09-17T07:08:08.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23323]: debug 2024-09-17T07:08:08.354+0000 7fddc745c700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-17T07:08:08.864 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:08 smithi044 podman[23561]: 2024-09-17 07:08:08.612190021 +0000 UTC m=+0.366005582 container died 3b36ced0728adfb2222457184c9bb2fde6f49b7bfc48a50017882cb777bfa86a (image=quay.io/ceph/ceph:v16.2.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-16.2.0, RELEASE=HEAD, GIT_CLEAN=True, maintainer=Dimitri Savineau , org.label-schema.name=CentOS Base Image, ceph=True, org.label-schema.build-date=20201204, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557) 2024-09-17T07:08:09.114 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:08 smithi044 podman[23561]: 2024-09-17 07:08:08.938004879 +0000 UTC m=+0.691820426 container cleanup 3b36ced0728adfb2222457184c9bb2fde6f49b7bfc48a50017882cb777bfa86a (image=quay.io/ceph/ceph:v16.2.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a, maintainer=Dimitri Savineau , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-16.2.0, RELEASE=HEAD, GIT_CLEAN=True, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, org.label-schema.vendor=CentOS) 2024-09-17T07:08:09.114 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:08 smithi044 podman[23561]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a 2024-09-17T07:08:09.114 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 podman[23575]: 2024-09-17 07:08:09.097320442 +0000 UTC m=+0.478596175 container remove 3b36ced0728adfb2222457184c9bb2fde6f49b7bfc48a50017882cb777bfa86a (image=quay.io/ceph/ceph:v16.2.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a, ceph=True, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=HEAD, maintainer=Dimitri Savineau , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-16.2.0, GIT_CLEAN=True, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-17T07:08:09.114 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-17T07:08:09.115 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 systemd[1]: Stopped Ceph mon.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:08:09.115 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 systemd[1]: Starting Ceph mon.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:08:09.544 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 podman[23618]: 2024-09-17 07:08:09.196458237 +0000 UTC m=+0.017527331 image pull 24ecd6d5f14c2268c5d2c666aa151456949cf76c6747d873c6fcf41f6f672d15 quay.io/ceph/ceph:v16.2.0 2024-09-17T07:08:09.545 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 podman[23618]: 2024-09-17 07:08:09.298021042 +0000 UTC m=+0.119090120 container create 7905efa25a24d56dd5270ce48fb2ec6b71b09f25f2272ad1322ce65bb8a3ebaf (image=quay.io/ceph/ceph:v16.2.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=-16.2.0, maintainer=Dimitri Savineau , org.label-schema.license=GPLv2, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Base Image, org.label-schema.schema-version=1.0, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20201204) 2024-09-17T07:08:09.545 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 podman[23618]: 2024-09-17 07:08:09.541608544 +0000 UTC m=+0.362677632 container init 7905efa25a24d56dd5270ce48fb2ec6b71b09f25f2272ad1322ce65bb8a3ebaf (image=quay.io/ceph/ceph:v16.2.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-16.2.0, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, ceph=True, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.build-date=20201204, maintainer=Dimitri Savineau , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Base Image) 2024-09-17T07:08:09.545 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 podman[23618]: 2024-09-17 07:08:09.544942345 +0000 UTC m=+0.366011424 container start 7905efa25a24d56dd5270ce48fb2ec6b71b09f25f2272ad1322ce65bb8a3ebaf (image=quay.io/ceph/ceph:v16.2.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a, org.label-schema.license=GPLv2, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.build-date=20201204, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-16.2.0, maintainer=Dimitri Savineau , org.label-schema.name=CentOS Base Image, GIT_CLEAN=True) 2024-09-17T07:08:09.645 INFO:teuthology.orchestra.run.smithi044.stderr:Setting mon public_network to 172.21.0.0/20 2024-09-17T07:08:09.645 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi044 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/817ae996-74c3-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmpboltjppk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpukm_xx4s:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 config set mon public_network 172.21.0.0/20 2024-09-17T07:08:09.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.564+0000 7f714a860700 0 set uid:gid to 167:167 (ceph:ceph) 2024-09-17T07:08:09.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.564+0000 7f714a860700 0 ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process ceph-mon, pid 2 2024-09-17T07:08:09.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.564+0000 7f714a860700 0 pidfile_write: ignore empty --pid-file 2024-09-17T07:08:09.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 0 load: jerasure load: lrc load: isa 2024-09-17T07:08:09.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: RocksDB version: 6.8.1 2024-09-17T07:08:09.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: 2024-09-17T07:08:09.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-17T07:08:09.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Compile date Mar 30 2021 2024-09-17T07:08:09.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: DB SUMMARY 2024-09-17T07:08:09.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: 2024-09-17T07:08:09.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: CURRENT file: CURRENT 2024-09-17T07:08:09.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: 2024-09-17T07:08:09.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: IDENTITY file: IDENTITY 2024-09-17T07:08:09.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: 2024-09-17T07:08:09.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-09-17T07:08:09.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: 2024-09-17T07:08:09.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000004.sst 2024-09-17T07:08:09.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: 2024-09-17T07:08:09.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000006.log size: 83080 ; 2024-09-17T07:08:09.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: 2024-09-17T07:08:09.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.error_if_exists: 0 2024-09-17T07:08:09.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.create_if_missing: 0 2024-09-17T07:08:09.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.paranoid_checks: 1 2024-09-17T07:08:09.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.env: 0x55df3199d1c0 2024-09-17T07:08:09.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.fs: Posix File System 2024-09-17T07:08:09.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.info_log: 0x55df33b72640 2024-09-17T07:08:09.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.max_file_opening_threads: 16 2024-09-17T07:08:09.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.statistics: (nil) 2024-09-17T07:08:09.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.use_fsync: 0 2024-09-17T07:08:09.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.max_log_file_size: 0 2024-09-17T07:08:09.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-17T07:08:09.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-09-17T07:08:09.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.keep_log_file_num: 1000 2024-09-17T07:08:09.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.recycle_log_file_num: 0 2024-09-17T07:08:09.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.allow_fallocate: 1 2024-09-17T07:08:09.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.allow_mmap_reads: 0 2024-09-17T07:08:09.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.allow_mmap_writes: 0 2024-09-17T07:08:09.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.use_direct_reads: 0 2024-09-17T07:08:09.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-17T07:08:09.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.create_missing_column_families: 0 2024-09-17T07:08:09.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.db_log_dir: 2024-09-17T07:08:09.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-09-17T07:08:09.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-09-17T07:08:09.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.max_subcompactions: 1 2024-09-17T07:08:09.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.max_background_flushes: -1 2024-09-17T07:08:09.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-09-17T07:08:09.830 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-09-17T07:08:09.830 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-17T07:08:09.831 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-17T07:08:09.831 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-09-17T07:08:09.831 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.advise_random_on_open: 1 2024-09-17T07:08:09.831 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.db_write_buffer_size: 0 2024-09-17T07:08:09.831 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.write_buffer_manager: 0x55df33c09e90 2024-09-17T07:08:09.831 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-17T07:08:09.831 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-17T07:08:09.831 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-17T07:08:09.831 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-09-17T07:08:09.831 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.rate_limiter: (nil) 2024-09-17T07:08:09.831 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-17T07:08:09.831 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.wal_recovery_mode: 2 2024-09-17T07:08:09.831 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.enable_thread_tracking: 0 2024-09-17T07:08:09.831 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.enable_pipelined_write: 0 2024-09-17T07:08:09.832 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.unordered_write: 0 2024-09-17T07:08:09.832 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-17T07:08:09.832 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-17T07:08:09.832 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-17T07:08:09.832 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-17T07:08:09.832 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.row_cache: None 2024-09-17T07:08:09.832 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.wal_filter: None 2024-09-17T07:08:09.832 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-17T07:08:09.832 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.allow_ingest_behind: 0 2024-09-17T07:08:09.832 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.preserve_deletes: 0 2024-09-17T07:08:09.832 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.two_write_queues: 0 2024-09-17T07:08:09.832 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.manual_wal_flush: 0 2024-09-17T07:08:09.832 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.atomic_flush: 0 2024-09-17T07:08:09.833 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-17T07:08:09.833 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-09-17T07:08:09.833 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-09-17T07:08:09.833 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.log_readahead_size: 0 2024-09-17T07:08:09.833 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-09-17T07:08:09.833 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.max_background_jobs: 2 2024-09-17T07:08:09.833 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.max_background_compactions: -1 2024-09-17T07:08:09.833 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-17T07:08:09.833 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-17T07:08:09.833 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-09-17T07:08:09.833 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.max_total_wal_size: 0 2024-09-17T07:08:09.833 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-17T07:08:09.833 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-09-17T07:08:09.834 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-09-17T07:08:09.834 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-17T07:08:09.834 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.max_open_files: -1 2024-09-17T07:08:09.834 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.bytes_per_sync: 0 2024-09-17T07:08:09.834 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-09-17T07:08:09.834 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-09-17T07:08:09.834 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Options.compaction_readahead_size: 0 2024-09-17T07:08:09.834 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Compression algorithms supported: 2024-09-17T07:08:09.834 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-17T07:08:09.834 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: kZSTD supported: 0 2024-09-17T07:08:09.834 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: kXpressCompression supported: 0 2024-09-17T07:08:09.834 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: kLZ4HCCompression supported: 1 2024-09-17T07:08:09.834 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: kLZ4Compression supported: 1 2024-09-17T07:08:09.835 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: kBZip2Compression supported: 0 2024-09-17T07:08:09.835 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: kZlibCompression supported: 1 2024-09-17T07:08:09.835 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: kSnappyCompression supported: 1 2024-09-17T07:08:09.835 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.568+0000 7f714a860700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-09-17T07:08:09.835 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000005 2024-09-17T07:08:09.835 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: 2024-09-17T07:08:09.835 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-09-17T07:08:09.835 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: 2024-09-17T07:08:09.835 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-17T07:08:09.835 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.merge_operator: 2024-09-17T07:08:09.835 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.compaction_filter: None 2024-09-17T07:08:09.835 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.compaction_filter_factory: None 2024-09-17T07:08:09.835 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-09-17T07:08:09.835 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-09-17T07:08:09.835 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55df33b180d8) 2024-09-17T07:08:09.836 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cache_index_and_filter_blocks: 1 2024-09-17T07:08:09.836 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-17T07:08:09.836 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-17T07:08:09.836 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: pin_top_level_index_and_filter: 1 2024-09-17T07:08:09.836 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: index_type: 0 2024-09-17T07:08:09.836 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: data_block_index_type: 0 2024-09-17T07:08:09.836 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: index_shortening: 1 2024-09-17T07:08:09.836 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: data_block_hash_table_util_ratio: 0.750000 2024-09-17T07:08:09.836 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: hash_index_allow_collision: 1 2024-09-17T07:08:09.836 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: checksum: 1 2024-09-17T07:08:09.836 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: no_block_cache: 0 2024-09-17T07:08:09.836 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: block_cache: 0x55df33b4ef10 2024-09-17T07:08:09.836 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: block_cache_name: BinnedLRUCache 2024-09-17T07:08:09.837 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: block_cache_options: 2024-09-17T07:08:09.837 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: capacity : 536870912 2024-09-17T07:08:09.837 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: num_shard_bits : 4 2024-09-17T07:08:09.837 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: strict_capacity_limit : 0 2024-09-17T07:08:09.837 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: high_pri_pool_ratio: 0.000 2024-09-17T07:08:09.837 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: block_cache_compressed: (nil) 2024-09-17T07:08:09.837 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: persistent_cache: (nil) 2024-09-17T07:08:09.837 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: block_size: 4096 2024-09-17T07:08:09.837 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: block_size_deviation: 10 2024-09-17T07:08:09.837 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: block_restart_interval: 16 2024-09-17T07:08:09.837 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: index_block_restart_interval: 1 2024-09-17T07:08:09.837 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: metadata_block_size: 4096 2024-09-17T07:08:09.837 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: partition_filters: 0 2024-09-17T07:08:09.837 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: use_delta_encoding: 1 2024-09-17T07:08:09.838 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-17T07:08:09.838 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: whole_key_filtering: 1 2024-09-17T07:08:09.838 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: verify_compression: 0 2024-09-17T07:08:09.838 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: read_amp_bytes_per_bit: 0 2024-09-17T07:08:09.838 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: format_version: 2 2024-09-17T07:08:09.838 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: enable_index_compression: 1 2024-09-17T07:08:09.838 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: block_align: 0 2024-09-17T07:08:09.838 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: 2024-09-17T07:08:09.838 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.write_buffer_size: 33554432 2024-09-17T07:08:09.838 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.max_write_buffer_number: 2 2024-09-17T07:08:09.838 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.compression: NoCompression 2024-09-17T07:08:09.838 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.bottommost_compression: Disabled 2024-09-17T07:08:09.838 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.prefix_extractor: nullptr 2024-09-17T07:08:09.838 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-17T07:08:09.839 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.num_levels: 7 2024-09-17T07:08:09.839 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-17T07:08:09.839 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-17T07:08:09.839 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-17T07:08:09.839 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-17T07:08:09.839 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-17T07:08:09.839 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-17T07:08:09.839 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-17T07:08:09.839 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-17T07:08:09.839 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-17T07:08:09.839 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-09-17T07:08:09.839 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.compression_opts.level: 32767 2024-09-17T07:08:09.839 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.compression_opts.strategy: 0 2024-09-17T07:08:09.839 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-17T07:08:09.840 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-17T07:08:09.840 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.compression_opts.enabled: false 2024-09-17T07:08:09.840 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-17T07:08:09.840 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-17T07:08:09.840 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-17T07:08:09.840 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.target_file_size_base: 67108864 2024-09-17T07:08:09.840 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-09-17T07:08:09.840 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-17T07:08:09.840 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-17T07:08:09.840 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-17T07:08:09.840 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-17T07:08:09.840 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-17T07:08:09.840 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-17T07:08:09.841 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-17T07:08:09.841 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-17T07:08:09.841 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-17T07:08:09.841 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-17T07:08:09.841 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-17T07:08:09.841 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-17T07:08:09.841 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.arena_block_size: 4194304 2024-09-17T07:08:09.841 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-17T07:08:09.841 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-17T07:08:09.841 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-17T07:08:09.841 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.disable_auto_compactions: 0 2024-09-17T07:08:09.841 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-17T07:08:09.841 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-17T07:08:09.842 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-17T07:08:09.842 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-17T07:08:09.842 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-17T07:08:09.844 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-17T07:08:09.844 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-17T07:08:09.844 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-17T07:08:09.844 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-17T07:08:09.844 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-17T07:08:09.844 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.table_properties_collectors: 2024-09-17T07:08:09.845 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.inplace_update_support: 0 2024-09-17T07:08:09.845 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-09-17T07:08:09.845 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-17T07:08:09.845 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-17T07:08:09.845 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-09-17T07:08:09.845 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.bloom_locality: 0 2024-09-17T07:08:09.845 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.max_successive_merges: 0 2024-09-17T07:08:09.845 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-09-17T07:08:09.845 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.paranoid_file_checks: 0 2024-09-17T07:08:09.845 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.force_consistency_checks: 0 2024-09-17T07:08:09.845 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.report_bg_io_stats: 0 2024-09-17T07:08:09.845 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.ttl: 2592000 2024-09-17T07:08:09.845 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-09-17T07:08:09.845 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-17T07:08:09.845 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: 2024-09-17T07:08:09.846 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.569+0000 7f714a860700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-09-17T07:08:09.846 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: 2024-09-17T07:08:09.846 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.570+0000 7f714a860700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726556889572038, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-09-17T07:08:09.846 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.570+0000 7f714a860700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-09-17T07:08:09.846 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.570+0000 7f714a860700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-17T07:08:09.846 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.571+0000 7f714a860700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726556889573303, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 80048, "table_properties": {"data_size": 78306, "index_size": 347, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 581, "raw_key_size": 8979, "raw_average_key_size": 46, "raw_value_size": 73065, "raw_average_value_size": 376, "num_data_blocks": 11, "num_entries": 194, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1726556889, "oldest_key_time": 3, "file_creation_time": 0}} 2024-09-17T07:08:09.846 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.571+0000 7f714a860700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-09-17T07:08:09.846 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: 2024-09-17T07:08:09.846 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.571+0000 7f714a860700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726556889573836, "job": 1, "event": "recovery_finished"} 2024-09-17T07:08:09.846 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.573+0000 7f714a860700 4 rocksdb: DB pointer 0x55df33c13800 2024-09-17T07:08:09.846 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.573+0000 7f713167a700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-09-17T07:08:09.846 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.573+0000 7f713167a700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-09-17T07:08:09.846 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: ** DB Stats ** 2024-09-17T07:08:09.846 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T07:08:09.847 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-17T07:08:09.847 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T07:08:09.847 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T07:08:09.847 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-17T07:08:09.847 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-17T07:08:09.847 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T07:08:09.847 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: 2024-09-17T07:08:09.847 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: ** Compaction Stats [default] ** 2024-09-17T07:08:09.847 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-17T07:08:09.847 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T07:08:09.847 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: L0 2/0 79.83 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 92.6 0.00 0.00 1 0.001 0 0 2024-09-17T07:08:09.847 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: Sum 2/0 79.83 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 92.6 0.00 0.00 1 0.001 0 0 2024-09-17T07:08:09.847 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 92.6 0.00 0.00 1 0.001 0 0 2024-09-17T07:08:09.848 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: 2024-09-17T07:08:09.848 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: ** Compaction Stats [default] ** 2024-09-17T07:08:09.848 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-17T07:08:09.848 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T07:08:09.848 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 92.6 0.00 0.00 1 0.001 0 0 2024-09-17T07:08:09.848 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T07:08:09.848 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: Flush(GB): cumulative 0.000, interval 0.000 2024-09-17T07:08:09.848 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T07:08:09.848 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T07:08:09.848 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T07:08:09.848 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: AddFile(Keys): cumulative 0, interval 0 2024-09-17T07:08:09.848 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: Cumulative compaction: 0.00 GB write, 18.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T07:08:09.848 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: Interval compaction: 0.00 GB write, 18.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T07:08:09.848 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T07:08:09.849 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: 2024-09-17T07:08:09.849 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: ** File Read Latency Histogram By Level [default] ** 2024-09-17T07:08:09.849 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: 2024-09-17T07:08:09.849 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: ** Compaction Stats [default] ** 2024-09-17T07:08:09.849 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-17T07:08:09.849 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T07:08:09.849 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: L0 2/0 79.83 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 92.6 0.00 0.00 1 0.001 0 0 2024-09-17T07:08:09.849 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: Sum 2/0 79.83 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 92.6 0.00 0.00 1 0.001 0 0 2024-09-17T07:08:09.849 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-17T07:08:09.849 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: 2024-09-17T07:08:09.849 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: ** Compaction Stats [default] ** 2024-09-17T07:08:09.849 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-17T07:08:09.849 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T07:08:09.849 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 92.6 0.00 0.00 1 0.001 0 0 2024-09-17T07:08:09.850 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T07:08:09.850 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: Flush(GB): cumulative 0.000, interval 0.000 2024-09-17T07:08:09.850 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T07:08:09.850 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T07:08:09.850 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T07:08:09.850 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: AddFile(Keys): cumulative 0, interval 0 2024-09-17T07:08:09.850 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: Cumulative compaction: 0.00 GB write, 18.41 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T07:08:09.850 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T07:08:09.850 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T07:08:09.850 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: 2024-09-17T07:08:09.850 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: ** File Read Latency Histogram By Level [default] ** 2024-09-17T07:08:09.850 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: 2024-09-17T07:08:09.850 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.574+0000 7f714a860700 0 starting mon.a rank 0 at public addrs [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] at bind addrs [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 817ae996-74c3-11ef-bceb-c7b262605968 2024-09-17T07:08:09.850 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.575+0000 7f714a860700 1 mon.a@-1(???) e1 preinit fsid 817ae996-74c3-11ef-bceb-c7b262605968 2024-09-17T07:08:09.851 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.575+0000 7f714a860700 0 mon.a@-1(???).mds e1 new map 2024-09-17T07:08:09.851 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.575+0000 7f714a860700 0 mon.a@-1(???).mds e1 print_map 2024-09-17T07:08:09.851 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: e1 2024-09-17T07:08:09.851 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: enable_multiple, ever_enabled_multiple: 1,1 2024-09-17T07:08:09.851 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-17T07:08:09.851 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: legacy client fscid: -1 2024-09-17T07:08:09.851 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: 2024-09-17T07:08:09.851 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: No filesystems configured 2024-09-17T07:08:09.851 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: 2024-09-17T07:08:09.852 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.576+0000 7f714a860700 0 mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-17T07:08:09.852 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.576+0000 7f714a860700 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T07:08:09.852 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.576+0000 7f714a860700 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T07:08:09.852 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.576+0000 7f714a860700 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T07:08:09.852 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:07.665257+0000 mon.a (mon.0) 10 : audit [DBG] from='client.? 172.21.15.44:0/1580534661' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:08:09.852 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:08:09.576+0000 7f714a860700 1 mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-17T07:08:09.852 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:08:09.582547+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-09-17T07:08:09.852 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:08:09.582614+0000 mon.a (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {a=[v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0]} 2024-09-17T07:08:09.852 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:08:09.582672+0000 mon.a (mon.0) 3 : cluster [DBG] fsmap 2024-09-17T07:08:09.852 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:08:09.582713+0000 mon.a (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-09-17T07:08:09.852 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:08:09.584663+0000 mon.a (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-09-17T07:08:09.852 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 bash[23618]: 7905efa25a24d56dd5270ce48fb2ec6b71b09f25f2272ad1322ce65bb8a3ebaf 2024-09-17T07:08:09.852 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:09 smithi044 systemd[1]: Started Ceph mon.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:08:10.627 INFO:teuthology.orchestra.run.smithi044.stderr:Wrote config to /etc/ceph/ceph.conf 2024-09-17T07:08:10.627 INFO:teuthology.orchestra.run.smithi044.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-17T07:08:10.627 INFO:teuthology.orchestra.run.smithi044.stderr:Creating mgr... 2024-09-17T07:08:10.627 INFO:teuthology.orchestra.run.smithi044.stderr:Verifying port 9283 ... 2024-09-17T07:08:10.629 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: systemctl daemon-reload 2024-09-17T07:08:10.787 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: systemctl stop ceph-817ae996-74c3-11ef-bceb-c7b262605968@mgr.y 2024-09-17T07:08:10.801 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: systemctl reset-failed ceph-817ae996-74c3-11ef-bceb-c7b262605968@mgr.y 2024-09-17T07:08:10.807 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: Failed to reset failed state of unit ceph-817ae996-74c3-11ef-bceb-c7b262605968@mgr.y.service: Unit ceph-817ae996-74c3-11ef-bceb-c7b262605968@mgr.y.service not loaded. 2024-09-17T07:08:10.808 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: systemctl enable ceph-817ae996-74c3-11ef-bceb-c7b262605968@mgr.y 2024-09-17T07:08:10.815 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: Created symlink /etc/systemd/system/ceph-817ae996-74c3-11ef-bceb-c7b262605968.target.wants/ceph-817ae996-74c3-11ef-bceb-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-817ae996-74c3-11ef-bceb-c7b262605968@.service. 2024-09-17T07:08:10.960 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: systemctl start ceph-817ae996-74c3-11ef-bceb-c7b262605968@mgr.y 2024-09-17T07:08:11.074 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:08:10 smithi044 systemd[1]: Starting Ceph mgr.y for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:08:11.460 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:10.136956+0000 mon.a (mon.0) 6 : audit [INF] from='client.? 172.21.15.44:0/2661411568' entity='client.admin' 2024-09-17T07:08:11.460 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:08:11 smithi044 podman[23829]: 2024-09-17 07:08:11.05079507 +0000 UTC m=+0.016117791 image pull 24ecd6d5f14c2268c5d2c666aa151456949cf76c6747d873c6fcf41f6f672d15 quay.io/ceph/ceph:v16.2.0 2024-09-17T07:08:11.460 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:08:11 smithi044 podman[23829]: 2024-09-17 07:08:11.165871643 +0000 UTC m=+0.131194365 container create 1b5bd0a0cb115ee80ee789f0c3215c98a38d30847177654cf9103e4a409ea40f (image=quay.io/ceph/ceph:v16.2.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.y, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=HEAD, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, org.label-schema.build-date=20201204, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-16.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Base Image, maintainer=Dimitri Savineau ) 2024-09-17T07:08:11.460 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:08:11 smithi044 podman[23829]: 2024-09-17 07:08:11.359397878 +0000 UTC m=+0.324720600 container init 1b5bd0a0cb115ee80ee789f0c3215c98a38d30847177654cf9103e4a409ea40f (image=quay.io/ceph/ceph:v16.2.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.y, RELEASE=HEAD, CEPH_POINT_RELEASE=-16.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, org.label-schema.build-date=20201204, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Base Image, org.label-schema.schema-version=1.0, maintainer=Dimitri Savineau , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-17T07:08:11.460 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:08:11 smithi044 podman[23829]: 2024-09-17 07:08:11.36242394 +0000 UTC m=+0.327746661 container start 1b5bd0a0cb115ee80ee789f0c3215c98a38d30847177654cf9103e4a409ea40f (image=quay.io/ceph/ceph:v16.2.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.y, org.label-schema.build-date=20201204, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-16.2.0, RELEASE=HEAD, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Dimitri Savineau , GIT_CLEAN=True) 2024-09-17T07:08:11.460 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:08:11 smithi044 bash[23829]: 1b5bd0a0cb115ee80ee789f0c3215c98a38d30847177654cf9103e4a409ea40f 2024-09-17T07:08:11.464 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: systemctl is-enabled firewalld.service 2024-09-17T07:08:11.474 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: disabled 2024-09-17T07:08:11.474 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: systemctl is-active firewalld.service 2024-09-17T07:08:11.485 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: inactive 2024-09-17T07:08:11.485 INFO:teuthology.orchestra.run.smithi044.stderr:firewalld.service is not enabled 2024-09-17T07:08:11.485 INFO:teuthology.orchestra.run.smithi044.stderr:Not possible to enable service . firewalld.service is not available 2024-09-17T07:08:11.485 INFO:teuthology.orchestra.run.smithi044.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-09-17T07:08:11.485 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: systemctl is-enabled firewalld.service 2024-09-17T07:08:11.490 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: disabled 2024-09-17T07:08:11.491 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: systemctl is-active firewalld.service 2024-09-17T07:08:11.497 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: inactive 2024-09-17T07:08:11.499 INFO:teuthology.orchestra.run.smithi044.stderr:firewalld.service is not enabled 2024-09-17T07:08:11.499 INFO:teuthology.orchestra.run.smithi044.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-09-17T07:08:11.499 INFO:teuthology.orchestra.run.smithi044.stderr:Waiting for mgr to start... 2024-09-17T07:08:11.499 INFO:teuthology.orchestra.run.smithi044.stderr:Waiting for mgr... 2024-09-17T07:08:11.499 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi044 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/817ae996-74c3-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmpboltjppk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpukm_xx4s:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 status -f json-pretty 2024-09-17T07:08:11.711 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:08:11 smithi044 systemd[1]: Started Ceph mgr.y for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:08:11.993 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: 2024-09-17T07:08:11.993 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: { 2024-09-17T07:08:11.993 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "fsid": "817ae996-74c3-11ef-bceb-c7b262605968", 2024-09-17T07:08:11.993 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "health": { 2024-09-17T07:08:11.993 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-09-17T07:08:11.993 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "checks": {}, 2024-09-17T07:08:11.993 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "mutes": [] 2024-09-17T07:08:11.994 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-09-17T07:08:11.994 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-09-17T07:08:11.994 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "quorum": [ 2024-09-17T07:08:11.994 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: 0 2024-09-17T07:08:11.994 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: ], 2024-09-17T07:08:11.994 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "quorum_names": [ 2024-09-17T07:08:11.994 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "a" 2024-09-17T07:08:11.994 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: ], 2024-09-17T07:08:11.994 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-09-17T07:08:11.994 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "monmap": { 2024-09-17T07:08:11.994 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T07:08:11.994 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-09-17T07:08:11.994 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_mons": 1 2024-09-17T07:08:11.994 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-09-17T07:08:11.995 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "osdmap": { 2024-09-17T07:08:11.995 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T07:08:11.995 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_osds": 0, 2024-09-17T07:08:11.995 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-09-17T07:08:11.995 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-09-17T07:08:11.995 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-09-17T07:08:11.998 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-09-17T07:08:11.998 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-09-17T07:08:11.998 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-09-17T07:08:11.998 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "pgmap": { 2024-09-17T07:08:11.999 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-09-17T07:08:11.999 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-09-17T07:08:11.999 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_pools": 0, 2024-09-17T07:08:11.999 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_objects": 0, 2024-09-17T07:08:11.999 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-09-17T07:08:11.999 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-09-17T07:08:11.999 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-09-17T07:08:11.999 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "bytes_total": 0 2024-09-17T07:08:11.999 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-09-17T07:08:11.999 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "fsmap": { 2024-09-17T07:08:11.999 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T07:08:11.999 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "by_rank": [], 2024-09-17T07:08:11.999 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "up:standby": 0 2024-09-17T07:08:12.000 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-09-17T07:08:12.000 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "mgrmap": { 2024-09-17T07:08:12.000 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "available": false, 2024-09-17T07:08:12.000 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-09-17T07:08:12.000 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "modules": [ 2024-09-17T07:08:12.000 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "iostat", 2024-09-17T07:08:12.000 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "restful" 2024-09-17T07:08:12.000 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: ], 2024-09-17T07:08:12.000 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "services": {} 2024-09-17T07:08:12.000 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-09-17T07:08:12.000 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "servicemap": { 2024-09-17T07:08:12.000 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T07:08:12.000 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "modified": "2024-09-17T07:08:04.408732+0000", 2024-09-17T07:08:12.000 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "services": {} 2024-09-17T07:08:12.001 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-09-17T07:08:12.001 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "progress_events": {} 2024-09-17T07:08:12.001 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: } 2024-09-17T07:08:12.282 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:11.989900+0000 mon.a (mon.0) 7 : audit [DBG] from='client.? 172.21.15.44:0/1162275443' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-17T07:08:12.756 INFO:teuthology.orchestra.run.smithi044.stderr:mgr not available, waiting (1/15)... 2024-09-17T07:08:14.758 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi044 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/817ae996-74c3-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmpboltjppk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpukm_xx4s:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 status -f json-pretty 2024-09-17T07:08:15.227 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: 2024-09-17T07:08:15.227 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: { 2024-09-17T07:08:15.227 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "fsid": "817ae996-74c3-11ef-bceb-c7b262605968", 2024-09-17T07:08:15.227 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "health": { 2024-09-17T07:08:15.227 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-09-17T07:08:15.227 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "checks": {}, 2024-09-17T07:08:15.228 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "mutes": [] 2024-09-17T07:08:15.228 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-09-17T07:08:15.228 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-09-17T07:08:15.228 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "quorum": [ 2024-09-17T07:08:15.228 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: 0 2024-09-17T07:08:15.228 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: ], 2024-09-17T07:08:15.228 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "quorum_names": [ 2024-09-17T07:08:15.228 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "a" 2024-09-17T07:08:15.228 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: ], 2024-09-17T07:08:15.228 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-09-17T07:08:15.228 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "monmap": { 2024-09-17T07:08:15.228 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T07:08:15.228 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-09-17T07:08:15.228 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_mons": 1 2024-09-17T07:08:15.229 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-09-17T07:08:15.229 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "osdmap": { 2024-09-17T07:08:15.229 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T07:08:15.229 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_osds": 0, 2024-09-17T07:08:15.229 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-09-17T07:08:15.229 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-09-17T07:08:15.229 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-09-17T07:08:15.232 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-09-17T07:08:15.232 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-09-17T07:08:15.232 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-09-17T07:08:15.233 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "pgmap": { 2024-09-17T07:08:15.233 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-09-17T07:08:15.233 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-09-17T07:08:15.233 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_pools": 0, 2024-09-17T07:08:15.233 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_objects": 0, 2024-09-17T07:08:15.233 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-09-17T07:08:15.233 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-09-17T07:08:15.233 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-09-17T07:08:15.233 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "bytes_total": 0 2024-09-17T07:08:15.233 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-09-17T07:08:15.233 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "fsmap": { 2024-09-17T07:08:15.233 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T07:08:15.233 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "by_rank": [], 2024-09-17T07:08:15.233 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "up:standby": 0 2024-09-17T07:08:15.234 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-09-17T07:08:15.234 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "mgrmap": { 2024-09-17T07:08:15.234 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "available": false, 2024-09-17T07:08:15.234 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-09-17T07:08:15.234 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "modules": [ 2024-09-17T07:08:15.234 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "iostat", 2024-09-17T07:08:15.234 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "restful" 2024-09-17T07:08:15.234 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: ], 2024-09-17T07:08:15.234 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "services": {} 2024-09-17T07:08:15.234 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-09-17T07:08:15.234 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "servicemap": { 2024-09-17T07:08:15.234 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T07:08:15.234 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "modified": "2024-09-17T07:08:04.408732+0000", 2024-09-17T07:08:15.234 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "services": {} 2024-09-17T07:08:15.234 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-09-17T07:08:15.235 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "progress_events": {} 2024-09-17T07:08:15.235 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: } 2024-09-17T07:08:15.758 INFO:teuthology.orchestra.run.smithi044.stderr:mgr not available, waiting (2/15)... 2024-09-17T07:08:15.885 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:15.224443+0000 mon.a (mon.0) 8 : audit [DBG] from='client.? 172.21.15.44:0/1114597845' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-17T07:08:17.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:08:16.487751+0000 mon.a (mon.0) 9 : cluster [INF] Activating manager daemon y 2024-09-17T07:08:17.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:08:16.490505+0000 mon.a (mon.0) 10 : cluster [DBG] mgrmap e2: y(active, starting, since 0.00284846s) 2024-09-17T07:08:17.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:16.491224+0000 mon.a (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.44:0/3994618094' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T07:08:17.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:16.491456+0000 mon.a (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.44:0/3994618094' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T07:08:17.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:16.491659+0000 mon.a (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.44:0/3994618094' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T07:08:17.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:16.491841+0000 mon.a (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.44:0/3994618094' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:08:17.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:16.492038+0000 mon.a (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.44:0/3994618094' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T07:08:17.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:08:16.493709+0000 mon.a (mon.0) 16 : cluster [INF] Manager daemon y is now available 2024-09-17T07:08:17.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:16.502357+0000 mon.a (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.44:0/3994618094' entity='mgr.y' 2024-09-17T07:08:17.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:16.505620+0000 mon.a (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.44:0/3994618094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:08:17.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:16.506726+0000 mon.a (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.44:0/3994618094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:08:17.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:16.512070+0000 mon.a (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.44:0/3994618094' entity='mgr.y' 2024-09-17T07:08:17.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:16.514687+0000 mon.a (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.44:0/3994618094' entity='mgr.y' 2024-09-17T07:08:17.760 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi044 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/817ae996-74c3-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmpboltjppk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpukm_xx4s:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 status -f json-pretty 2024-09-17T07:08:18.424 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: 2024-09-17T07:08:18.424 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: { 2024-09-17T07:08:18.425 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "fsid": "817ae996-74c3-11ef-bceb-c7b262605968", 2024-09-17T07:08:18.425 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "health": { 2024-09-17T07:08:18.425 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-09-17T07:08:18.425 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "checks": {}, 2024-09-17T07:08:18.425 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "mutes": [] 2024-09-17T07:08:18.425 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-09-17T07:08:18.425 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-09-17T07:08:18.425 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "quorum": [ 2024-09-17T07:08:18.425 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: 0 2024-09-17T07:08:18.425 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: ], 2024-09-17T07:08:18.425 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "quorum_names": [ 2024-09-17T07:08:18.425 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "a" 2024-09-17T07:08:18.426 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: ], 2024-09-17T07:08:18.426 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "quorum_age": 8, 2024-09-17T07:08:18.426 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "monmap": { 2024-09-17T07:08:18.426 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T07:08:18.426 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-09-17T07:08:18.426 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_mons": 1 2024-09-17T07:08:18.426 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-09-17T07:08:18.426 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "osdmap": { 2024-09-17T07:08:18.426 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T07:08:18.426 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_osds": 0, 2024-09-17T07:08:18.426 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-09-17T07:08:18.426 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-09-17T07:08:18.427 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-09-17T07:08:18.427 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-09-17T07:08:18.427 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-09-17T07:08:18.427 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-09-17T07:08:18.427 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "pgmap": { 2024-09-17T07:08:18.427 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-09-17T07:08:18.427 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-09-17T07:08:18.427 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_pools": 0, 2024-09-17T07:08:18.427 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_objects": 0, 2024-09-17T07:08:18.427 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-09-17T07:08:18.428 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-09-17T07:08:18.428 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-09-17T07:08:18.428 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "bytes_total": 0 2024-09-17T07:08:18.428 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-09-17T07:08:18.428 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "fsmap": { 2024-09-17T07:08:18.428 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T07:08:18.428 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "by_rank": [], 2024-09-17T07:08:18.428 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "up:standby": 0 2024-09-17T07:08:18.428 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-09-17T07:08:18.428 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "mgrmap": { 2024-09-17T07:08:18.428 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "available": true, 2024-09-17T07:08:18.428 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-09-17T07:08:18.428 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "modules": [ 2024-09-17T07:08:18.429 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "iostat", 2024-09-17T07:08:18.429 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "restful" 2024-09-17T07:08:18.429 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: ], 2024-09-17T07:08:18.429 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "services": {} 2024-09-17T07:08:18.429 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-09-17T07:08:18.429 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "servicemap": { 2024-09-17T07:08:18.429 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T07:08:18.429 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "modified": "2024-09-17T07:08:04.408732+0000", 2024-09-17T07:08:18.429 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "services": {} 2024-09-17T07:08:18.429 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: }, 2024-09-17T07:08:18.429 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "progress_events": {} 2024-09-17T07:08:18.429 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: } 2024-09-17T07:08:18.694 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:08:17.494221+0000 mon.a (mon.0) 22 : cluster [DBG] mgrmap e3: y(active, since 1.00656s) 2024-09-17T07:08:18.694 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:18.419227+0000 mon.a (mon.0) 23 : audit [DBG] from='client.? 172.21.15.44:0/4002702427' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-17T07:08:18.964 INFO:teuthology.orchestra.run.smithi044.stderr:mgr is available 2024-09-17T07:08:18.964 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi044 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/817ae996-74c3-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmpboltjppk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpukm_xx4s:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 config-key set mgr/cephadm/ssh_user root 2024-09-17T07:08:19.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:08:18.494716+0000 mon.a (mon.0) 24 : cluster [DBG] mgrmap e4: y(active, since 2s) 2024-09-17T07:08:19.630 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: set mgr/cephadm/ssh_user 2024-09-17T07:08:20.245 INFO:teuthology.orchestra.run.smithi044.stderr:Enabling cephadm module... 2024-09-17T07:08:20.245 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi044 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/817ae996-74c3-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmpboltjppk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpukm_xx4s:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 mgr module enable cephadm 2024-09-17T07:08:21.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:19.627953+0000 mon.a (mon.0) 25 : audit [INF] from='client.? 172.21.15.44:0/231441515' entity='client.admin' 2024-09-17T07:08:21.906 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:21 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:20.868498+0000 mon.a (mon.0) 26 : audit [INF] from='client.? 172.21.15.44:0/2820781382' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-17T07:08:21.906 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:08:21 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.y[23844]: ignoring --setuser ceph since I am not root 2024-09-17T07:08:21.907 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:08:21 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.y[23844]: ignoring --setgroup ceph since I am not root 2024-09-17T07:08:22.202 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi044 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/817ae996-74c3-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmpboltjppk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpukm_xx4s:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 mgr stat 2024-09-17T07:08:22.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:22 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:21.632646+0000 mon.a (mon.0) 27 : audit [INF] from='client.? 172.21.15.44:0/2820781382' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-17T07:08:22.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:22 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:08:21.632713+0000 mon.a (mon.0) 28 : cluster [DBG] mgrmap e5: y(active, since 5s) 2024-09-17T07:08:22.827 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: { 2024-09-17T07:08:22.827 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "epoch": 5, 2024-09-17T07:08:22.827 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "available": true, 2024-09-17T07:08:22.827 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "active_name": "y", 2024-09-17T07:08:22.827 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_standby": 0 2024-09-17T07:08:22.827 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: } 2024-09-17T07:08:23.440 INFO:teuthology.orchestra.run.smithi044.stderr:Waiting for the mgr to restart... 2024-09-17T07:08:23.441 INFO:teuthology.orchestra.run.smithi044.stderr:Waiting for mgr epoch 5... 2024-09-17T07:08:23.441 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi044 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/817ae996-74c3-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmpboltjppk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpukm_xx4s:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 tell mgr mgr_status 2024-09-17T07:08:23.690 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:23 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:22.826997+0000 mon.a (mon.0) 29 : audit [DBG] from='client.? 172.21.15.44:0/1943213996' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-17T07:08:27.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:08:26.744172+0000 mon.a (mon.0) 30 : cluster [INF] Active manager daemon y restarted 2024-09-17T07:08:27.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:08:26.744551+0000 mon.a (mon.0) 31 : cluster [INF] Activating manager daemon y 2024-09-17T07:08:27.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:08:26.746722+0000 mon.a (mon.0) 32 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-09-17T07:08:27.812 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: { 2024-09-17T07:08:27.812 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-09-17T07:08:27.812 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "initialized": true 2024-09-17T07:08:27.812 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: } 2024-09-17T07:08:28.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:08:26.796914+0000 mon.a (mon.0) 33 : cluster [DBG] mgrmap e6: y(active, starting, since 0.0524297s) 2024-09-17T07:08:28.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:26.799269+0000 mon.a (mon.0) 34 : audit [DBG] from='mgr.14116 172.21.15.44:0/2289010230' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:08:28.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:26.799941+0000 mon.a (mon.0) 35 : audit [DBG] from='mgr.14116 172.21.15.44:0/2289010230' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T07:08:28.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:26.800598+0000 mon.a (mon.0) 36 : audit [DBG] from='mgr.14116 172.21.15.44:0/2289010230' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T07:08:28.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:26.800756+0000 mon.a (mon.0) 37 : audit [DBG] from='mgr.14116 172.21.15.44:0/2289010230' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T07:08:28.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:26.800893+0000 mon.a (mon.0) 38 : audit [DBG] from='mgr.14116 172.21.15.44:0/2289010230' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T07:08:28.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:08:26.802841+0000 mon.a (mon.0) 39 : cluster [INF] Manager daemon y is now available 2024-09-17T07:08:28.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:26.816089+0000 mon.a (mon.0) 40 : audit [INF] from='mgr.14116 172.21.15.44:0/2289010230' entity='mgr.y' 2024-09-17T07:08:28.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:26.818617+0000 mon.a (mon.0) 41 : audit [INF] from='mgr.14116 172.21.15.44:0/2289010230' entity='mgr.y' 2024-09-17T07:08:28.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:26.821174+0000 mon.a (mon.0) 42 : audit [INF] from='mgr.14116 172.21.15.44:0/2289010230' entity='mgr.y' 2024-09-17T07:08:28.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:26.918118+0000 mon.a (mon.0) 43 : audit [INF] from='mgr.14116 172.21.15.44:0/2289010230' entity='mgr.y' 2024-09-17T07:08:28.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:26.918755+0000 mon.a (mon.0) 44 : audit [DBG] from='mgr.14116 172.21.15.44:0/2289010230' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:08:28.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:26.919732+0000 mon.a (mon.0) 45 : audit [DBG] from='mgr.14116 172.21.15.44:0/2289010230' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:08:28.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:26.921013+0000 mon.a (mon.0) 46 : audit [DBG] from='mgr.14116 172.21.15.44:0/2289010230' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:08:28.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:26.931978+0000 mon.a (mon.0) 47 : audit [INF] from='mgr.14116 172.21.15.44:0/2289010230' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:08:28.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:26.933237+0000 mon.a (mon.0) 48 : audit [INF] from='mgr.14116 172.21.15.44:0/2289010230' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:08:28.323 INFO:teuthology.orchestra.run.smithi044.stderr:mgr epoch 5 is available 2024-09-17T07:08:28.323 INFO:teuthology.orchestra.run.smithi044.stderr:Setting orchestrator backend to cephadm... 2024-09-17T07:08:28.323 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi044 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/817ae996-74c3-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmpboltjppk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpukm_xx4s:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 orch set backend cephadm 2024-09-17T07:08:28.990 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:28 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:27.803966+0000 mgr.y (mgr.14116) 1 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-17T07:08:28.990 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:28 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:08:27.805068+0000 mon.a (mon.0) 49 : cluster [DBG] mgrmap e7: y(active, since 1.06058s) 2024-09-17T07:08:28.990 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:28 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:27.811214+0000 mgr.y (mgr.14116) 2 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-17T07:08:29.486 INFO:teuthology.orchestra.run.smithi044.stderr:Generating ssh key... 2024-09-17T07:08:29.486 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi044 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/817ae996-74c3-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmpboltjppk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpukm_xx4s:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 cephadm generate-key 2024-09-17T07:08:30.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:08:28.806204+0000 mon.a (mon.0) 50 : cluster [DBG] mgrmap e8: y(active, since 2s) 2024-09-17T07:08:30.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:28.972999+0000 mgr.y (mgr.14116) 3 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:08:30.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:28.976815+0000 mon.a (mon.0) 51 : audit [INF] from='mgr.14116 172.21.15.44:0/2289010230' entity='mgr.y' 2024-09-17T07:08:30.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:28.980053+0000 mon.a (mon.0) 52 : audit [DBG] from='mgr.14116 172.21.15.44:0/2289010230' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:08:30.464 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:08:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.y[23844]: Generating public/private rsa key pair. 2024-09-17T07:08:30.464 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:08:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.y[23844]: Your identification has been saved in /tmp/tmp8gaxiw0e/key. 2024-09-17T07:08:30.464 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:08:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.y[23844]: Your public key has been saved in /tmp/tmp8gaxiw0e/key.pub. 2024-09-17T07:08:30.464 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:08:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.y[23844]: The key fingerprint is: 2024-09-17T07:08:30.465 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:08:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.y[23844]: SHA256:Pi5uB+kBGqWw7Ho9Cuwuv++8ZlM7tQTSJtZeRTl4Bw0 ceph-817ae996-74c3-11ef-bceb-c7b262605968 2024-09-17T07:08:30.465 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:08:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.y[23844]: The key's randomart image is: 2024-09-17T07:08:30.465 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:08:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.y[23844]: +---[RSA 3072]----+ 2024-09-17T07:08:30.465 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:08:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.y[23844]: | oE= | 2024-09-17T07:08:30.465 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:08:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.y[23844]: |. . . = o | 2024-09-17T07:08:30.465 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:08:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.y[23844]: |.o o o o o | 2024-09-17T07:08:30.465 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:08:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.y[23844]: |..o = = . | 2024-09-17T07:08:30.465 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:08:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.y[23844]: |. + * +S | 2024-09-17T07:08:30.465 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:08:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.y[23844]: |... *.o | 2024-09-17T07:08:30.465 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:08:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.y[23844]: |o. . o *o. | 2024-09-17T07:08:30.465 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:08:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.y[23844]: |+...* *.o. | 2024-09-17T07:08:30.465 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:08:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.y[23844]: |o=+B=*.+. | 2024-09-17T07:08:30.465 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:08:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.y[23844]: +----[SHA256]-----+ 2024-09-17T07:08:30.743 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi044 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/817ae996-74c3-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmpboltjppk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpukm_xx4s:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 cephadm get-pub-key 2024-09-17T07:08:31.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:08:30.060518+0000 mgr.y (mgr.14116) 5 : cephadm [INF] Generating ssh key... 2024-09-17T07:08:31.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:30.165734+0000 mon.a (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.44:0/2289010230' entity='mgr.y' 2024-09-17T07:08:31.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:30.168027+0000 mon.a (mon.0) 54 : audit [INF] from='mgr.14116 172.21.15.44:0/2289010230' entity='mgr.y' 2024-09-17T07:08:31.346 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSfvIA4wvxb6Ujn1G3x6pvF3+OKguJQoUpNH5iCO1cqRg23SgTrSU4ZrPLDH8qFJ2y+2nFH2DBzohs1SCUcNQBQ5EQvxD9/vZW2JwiCBd4gbK/RZhXt/SB31NKN9UVJt0CVXhzDOqtA97qXdIFQZ7qxgDiCSgWqKnsYxupTTDkwo9+C8NBrcO+g+JitITFfVEqEZofa7DQ5mOwZKFCkCN7fLKDzapuxcwtgOg+4tOae4HlxUkxMSfdBEj0TyNO3v2CE97O8hFXSulNmW9X70wBugUysQp2xu71hmStuWUDZ1AafXCbHgMlUjSNaHVNLWTyNyUODSST+AcUyROgr5nWWGNpQEgz49BeYyjr7j6ENHtUZm1RJQMX7Cduz0Nrjp+jbzgjGTUE3MY2BlToXX1amk01o8JUvc5Qgsty2AcOA74AijC0V799uTpLStX30TGldP/iCz+vTMmf3SeheD/irdOjiimf+PquT3FFjy3vivB9Q3LZkwiwsbT6hNSbbB8= ceph-817ae996-74c3-11ef-bceb-c7b262605968 2024-09-17T07:08:31.881 INFO:teuthology.orchestra.run.smithi044.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-17T07:08:31.881 INFO:teuthology.orchestra.run.smithi044.stderr:Adding key to root@localhost authorized_keys... 2024-09-17T07:08:31.890 INFO:teuthology.orchestra.run.smithi044.stderr:Adding host smithi044... 2024-09-17T07:08:31.890 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi044 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/817ae996-74c3-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmpboltjppk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpukm_xx4s:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 orch host add smithi044 2024-09-17T07:08:32.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:32 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:31.345511+0000 mgr.y (mgr.14116) 6 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:08:32.689 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:08:32 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.y[23844]: Warning: Permanently added 'smithi044,172.21.15.44' (ECDSA) to the list of known hosts. 2024-09-17T07:08:33.111 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: Added host 'smithi044' 2024-09-17T07:08:33.443 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:32.482324+0000 mgr.y (mgr.14116) 7 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi044", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:08:33.443 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:33.109131+0000 mon.a (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.44:0/2289010230' entity='mgr.y' 2024-09-17T07:08:33.443 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:33.109688+0000 mon.a (mon.0) 56 : audit [DBG] from='mgr.14116 172.21.15.44:0/2289010230' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:08:33.704 INFO:teuthology.orchestra.run.smithi044.stderr:Deploying unmanaged mon service... 2024-09-17T07:08:33.704 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi044 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/817ae996-74c3-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmpboltjppk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpukm_xx4s:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 orch apply mon --unmanaged 2024-09-17T07:08:34.256 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:08:33.109442+0000 mgr.y (mgr.14116) 8 : cephadm [INF] Added host smithi044 2024-09-17T07:08:34.432 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: Scheduled mon update... 2024-09-17T07:08:35.691 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:34.427765+0000 mgr.y (mgr.14116) 9 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:08:35.691 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:08:34.428509+0000 mgr.y (mgr.14116) 10 : cephadm [INF] Saving service mon spec with placement count:5 2024-09-17T07:08:35.691 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:34.431432+0000 mon.a (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.44:0/2289010230' entity='mgr.y' 2024-09-17T07:08:35.977 INFO:teuthology.orchestra.run.smithi044.stderr:Deploying unmanaged mgr service... 2024-09-17T07:08:35.977 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi044 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/817ae996-74c3-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmpboltjppk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpukm_xx4s:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 orch apply mgr --unmanaged 2024-09-17T07:08:36.618 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-09-17T07:08:37.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:35.593486+0000 mon.a (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.44:0/2289010230' entity='mgr.y' 2024-09-17T07:08:37.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:35.804304+0000 mon.a (mon.0) 59 : audit [INF] from='mgr.14116 172.21.15.44:0/2289010230' entity='mgr.y' 2024-09-17T07:08:37.784 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:36.612033+0000 mgr.y (mgr.14116) 11 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:08:37.784 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:08:36.612542+0000 mgr.y (mgr.14116) 12 : cephadm [INF] Saving service mgr spec with placement count:2 2024-09-17T07:08:37.784 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:36.617423+0000 mon.a (mon.0) 60 : audit [INF] from='mgr.14116 172.21.15.44:0/2289010230' entity='mgr.y' 2024-09-17T07:08:37.784 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:37.145687+0000 mon.a (mon.0) 61 : audit [INF] from='mgr.14116 172.21.15.44:0/2289010230' entity='mgr.y' 2024-09-17T07:08:37.786 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi044 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/817ae996-74c3-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmpboltjppk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpukm_xx4s:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 config set mgr mgr/cephadm/container_init True --force 2024-09-17T07:08:39.595 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:38.342639+0000 mon.a (mon.0) 62 : audit [INF] from='client.? 172.21.15.44:0/2731003132' entity='client.admin' 2024-09-17T07:08:39.892 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi044 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/817ae996-74c3-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmpboltjppk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpukm_xx4s:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 config set mgr mgr/dashboard/ssl_server_port 8443 2024-09-17T07:08:41.113 INFO:teuthology.orchestra.run.smithi044.stderr:Enabling the dashboard module... 2024-09-17T07:08:41.114 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi044 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/817ae996-74c3-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmpboltjppk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpukm_xx4s:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 mgr module enable dashboard 2024-09-17T07:08:41.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:41 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:40.475705+0000 mon.a (mon.0) 63 : audit [INF] from='client.? 172.21.15.44:0/3456571473' entity='client.admin' 2024-09-17T07:08:42.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:41.740332+0000 mon.a (mon.0) 64 : audit [INF] from='client.? 172.21.15.44:0/3692462821' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-17T07:08:42.824 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:08:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.y[23844]: ignoring --setuser ceph since I am not root 2024-09-17T07:08:42.825 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:08:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.y[23844]: ignoring --setgroup ceph since I am not root 2024-09-17T07:08:43.270 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi044 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/817ae996-74c3-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmpboltjppk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpukm_xx4s:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 mgr stat 2024-09-17T07:08:43.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:43 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:42.480016+0000 mon.a (mon.0) 65 : audit [INF] from='client.? 172.21.15.44:0/3692462821' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-17T07:08:43.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:43 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:08:42.480083+0000 mon.a (mon.0) 66 : cluster [DBG] mgrmap e9: y(active, since 15s) 2024-09-17T07:08:44.006 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: { 2024-09-17T07:08:44.006 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "epoch": 9, 2024-09-17T07:08:44.006 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "available": true, 2024-09-17T07:08:44.006 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "active_name": "y", 2024-09-17T07:08:44.006 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "num_standby": 0 2024-09-17T07:08:44.006 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: } 2024-09-17T07:08:44.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:44.005394+0000 mon.a (mon.0) 67 : audit [DBG] from='client.? 172.21.15.44:0/4181088117' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-17T07:08:44.585 INFO:teuthology.orchestra.run.smithi044.stderr:Waiting for the mgr to restart... 2024-09-17T07:08:44.585 INFO:teuthology.orchestra.run.smithi044.stderr:Waiting for mgr epoch 9... 2024-09-17T07:08:44.586 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi044 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/817ae996-74c3-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmpboltjppk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpukm_xx4s:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 tell mgr mgr_status 2024-09-17T07:08:48.324 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:08:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.y[23844]: Warning: Permanently added 'smithi044,172.21.15.44' (ECDSA) to the list of known hosts. 2024-09-17T07:08:48.738 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: { 2024-09-17T07:08:48.738 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-09-17T07:08:48.738 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: "initialized": true 2024-09-17T07:08:48.738 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: } 2024-09-17T07:08:48.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:08:47.676567+0000 mon.a (mon.0) 68 : cluster [INF] Active manager daemon y restarted 2024-09-17T07:08:48.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:08:47.677002+0000 mon.a (mon.0) 69 : cluster [INF] Activating manager daemon y 2024-09-17T07:08:48.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:08:47.678539+0000 mon.a (mon.0) 70 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-09-17T07:08:48.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:08:47.729170+0000 mon.a (mon.0) 71 : cluster [DBG] mgrmap e10: y(active, starting, since 0.052292s) 2024-09-17T07:08:48.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:47.729952+0000 mon.a (mon.0) 72 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:08:48.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:47.730675+0000 mon.a (mon.0) 73 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T07:08:48.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:47.731545+0000 mon.a (mon.0) 74 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T07:08:48.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:47.731686+0000 mon.a (mon.0) 75 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T07:08:48.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:47.731807+0000 mon.a (mon.0) 76 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T07:08:48.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:08:47.733684+0000 mon.a (mon.0) 77 : cluster [INF] Manager daemon y is now available 2024-09-17T07:08:48.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:47.839686+0000 mon.a (mon.0) 78 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:08:48.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:47.841157+0000 mon.a (mon.0) 79 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:08:48.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:47.862023+0000 mon.a (mon.0) 80 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:08:48.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:47.863626+0000 mon.a (mon.0) 81 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:08:49.622 INFO:teuthology.orchestra.run.smithi044.stderr:mgr epoch 9 is available 2024-09-17T07:08:49.622 INFO:teuthology.orchestra.run.smithi044.stderr:Generating a dashboard self-signed certificate... 2024-09-17T07:08:49.622 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi044 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/817ae996-74c3-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmpboltjppk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpukm_xx4s:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 dashboard create-self-signed-cert 2024-09-17T07:08:49.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:48.731021+0000 mgr.y (mgr.14144) 1 : audit [DBG] from='client.14148 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-17T07:08:49.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:08:48.731659+0000 mon.a (mon.0) 82 : cluster [DBG] mgrmap e11: y(active, since 1.05478s) 2024-09-17T07:08:49.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:48.737200+0000 mgr.y (mgr.14144) 2 : audit [DBG] from='client.14148 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-17T07:08:49.870 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:49.216044+0000 mon.a (mon.0) 83 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:08:50.466 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: Self-signed certificate created 2024-09-17T07:08:51.816 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:50.264270+0000 mgr.y (mgr.14144) 3 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:08:51.816 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:50.464497+0000 mon.a (mon.0) 84 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:08:51.816 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:50.466166+0000 mon.a (mon.0) 85 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:08:51.816 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:08:50.590647+0000 mon.a (mon.0) 86 : cluster [DBG] mgrmap e12: y(active, since 2s) 2024-09-17T07:08:51.818 INFO:teuthology.orchestra.run.smithi044.stderr:Creating initial admin user... 2024-09-17T07:08:51.819 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi044 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/817ae996-74c3-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmpboltjppk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpukm_xx4s:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp3ldjxfg7:/tmp/dashboard.pw:z quay.io/ceph/ceph:v16.2.0 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2024-09-17T07:08:52.668 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$G9DCf0jdmhG7AK4PD00ceOaREUaadeA2LpNJ7G0Eio2JHI537N3c6", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726556932, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-17T07:08:53.047 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:52.423595+0000 mgr.y (mgr.14144) 4 : audit [DBG] from='client.14156 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:08:53.342 INFO:teuthology.orchestra.run.smithi044.stderr:Fetching dashboard port number... 2024-09-17T07:08:53.343 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi044 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/817ae996-74c3-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmpboltjppk:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpukm_xx4s:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 config get mgr mgr/dashboard/ssl_server_port 2024-09-17T07:08:53.896 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:53 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:52.667862+0000 mon.a (mon.0) 87 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:08:53.896 INFO:teuthology.orchestra.run.smithi044.stderr:/usr/bin/ceph: 8443 2024-09-17T07:08:54.507 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: systemctl is-enabled firewalld.service 2024-09-17T07:08:54.517 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: disabled 2024-09-17T07:08:54.518 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: systemctl is-active firewalld.service 2024-09-17T07:08:54.525 INFO:teuthology.orchestra.run.smithi044.stderr:systemctl: inactive 2024-09-17T07:08:54.526 INFO:teuthology.orchestra.run.smithi044.stderr:firewalld.service is not enabled 2024-09-17T07:08:54.526 INFO:teuthology.orchestra.run.smithi044.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-17T07:08:54.529 INFO:teuthology.orchestra.run.smithi044.stderr:Ceph Dashboard is now available at: 2024-09-17T07:08:54.529 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-09-17T07:08:54.530 INFO:teuthology.orchestra.run.smithi044.stderr: URL: https://smithi044.front.sepia.ceph.com:8443/ 2024-09-17T07:08:54.530 INFO:teuthology.orchestra.run.smithi044.stderr: User: admin 2024-09-17T07:08:54.530 INFO:teuthology.orchestra.run.smithi044.stderr: Password: c786rmbgvx 2024-09-17T07:08:54.530 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-09-17T07:08:54.530 INFO:teuthology.orchestra.run.smithi044.stderr:You can access the Ceph CLI with: 2024-09-17T07:08:54.530 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-09-17T07:08:54.530 INFO:teuthology.orchestra.run.smithi044.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-17T07:08:54.530 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-09-17T07:08:54.530 INFO:teuthology.orchestra.run.smithi044.stderr:Please consider enabling telemetry to help improve Ceph: 2024-09-17T07:08:54.530 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-09-17T07:08:54.530 INFO:teuthology.orchestra.run.smithi044.stderr: ceph telemetry on 2024-09-17T07:08:54.530 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-09-17T07:08:54.530 INFO:teuthology.orchestra.run.smithi044.stderr:For more information see: 2024-09-17T07:08:54.531 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-09-17T07:08:54.531 INFO:teuthology.orchestra.run.smithi044.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2024-09-17T07:08:54.531 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-09-17T07:08:54.531 INFO:teuthology.orchestra.run.smithi044.stderr:Bootstrap complete. 2024-09-17T07:08:54.531 INFO:teuthology.orchestra.run.smithi044.stderr:Releasing lock 140421386999840 on /run/cephadm/817ae996-74c3-11ef-bceb-c7b262605968.lock 2024-09-17T07:08:54.531 INFO:teuthology.orchestra.run.smithi044.stderr:Lock 140421386999840 released on /run/cephadm/817ae996-74c3-11ef-bceb-c7b262605968.lock 2024-09-17T07:08:54.567 INFO:tasks.cephadm:Fetching config... 2024-09-17T07:08:54.568 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-09-17T07:08:54.568 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-17T07:08:54.587 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-17T07:08:54.587 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-09-17T07:08:54.587 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-17T07:08:54.644 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-17T07:08:54.644 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-09-17T07:08:54.644 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/var/lib/ceph/817ae996-74c3-11ef-bceb-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-17T07:08:54.713 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-17T07:08:54.713 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-09-17T07:08:54.714 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-17T07:08:54.769 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-17T07:08:54.770 DEBUG:teuthology.orchestra.run.smithi044:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSfvIA4wvxb6Ujn1G3x6pvF3+OKguJQoUpNH5iCO1cqRg23SgTrSU4ZrPLDH8qFJ2y+2nFH2DBzohs1SCUcNQBQ5EQvxD9/vZW2JwiCBd4gbK/RZhXt/SB31NKN9UVJt0CVXhzDOqtA97qXdIFQZ7qxgDiCSgWqKnsYxupTTDkwo9+C8NBrcO+g+JitITFfVEqEZofa7DQ5mOwZKFCkCN7fLKDzapuxcwtgOg+4tOae4HlxUkxMSfdBEj0TyNO3v2CE97O8hFXSulNmW9X70wBugUysQp2xu71hmStuWUDZ1AafXCbHgMlUjSNaHVNLWTyNyUODSST+AcUyROgr5nWWGNpQEgz49BeYyjr7j6ENHtUZm1RJQMX7Cduz0Nrjp+jbzgjGTUE3MY2BlToXX1amk01o8JUvc5Qgsty2AcOA74AijC0V799uTpLStX30TGldP/iCz+vTMmf3SeheD/irdOjiimf+PquT3FFjy3vivB9Q3LZkwiwsbT6hNSbbB8= ceph-817ae996-74c3-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-17T07:08:54.822 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:53.895908+0000 mon.a (mon.0) 88 : audit [DBG] from='client.? 172.21.15.44:0/3497307366' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-17T07:08:54.846 INFO:teuthology.orchestra.run.smithi044.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSfvIA4wvxb6Ujn1G3x6pvF3+OKguJQoUpNH5iCO1cqRg23SgTrSU4ZrPLDH8qFJ2y+2nFH2DBzohs1SCUcNQBQ5EQvxD9/vZW2JwiCBd4gbK/RZhXt/SB31NKN9UVJt0CVXhzDOqtA97qXdIFQZ7qxgDiCSgWqKnsYxupTTDkwo9+C8NBrcO+g+JitITFfVEqEZofa7DQ5mOwZKFCkCN7fLKDzapuxcwtgOg+4tOae4HlxUkxMSfdBEj0TyNO3v2CE97O8hFXSulNmW9X70wBugUysQp2xu71hmStuWUDZ1AafXCbHgMlUjSNaHVNLWTyNyUODSST+AcUyROgr5nWWGNpQEgz49BeYyjr7j6ENHtUZm1RJQMX7Cduz0Nrjp+jbzgjGTUE3MY2BlToXX1amk01o8JUvc5Qgsty2AcOA74AijC0V799uTpLStX30TGldP/iCz+vTMmf3SeheD/irdOjiimf+PquT3FFjy3vivB9Q3LZkwiwsbT6hNSbbB8= ceph-817ae996-74c3-11ef-bceb-c7b262605968 2024-09-17T07:08:54.858 DEBUG:teuthology.orchestra.run.smithi179:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSfvIA4wvxb6Ujn1G3x6pvF3+OKguJQoUpNH5iCO1cqRg23SgTrSU4ZrPLDH8qFJ2y+2nFH2DBzohs1SCUcNQBQ5EQvxD9/vZW2JwiCBd4gbK/RZhXt/SB31NKN9UVJt0CVXhzDOqtA97qXdIFQZ7qxgDiCSgWqKnsYxupTTDkwo9+C8NBrcO+g+JitITFfVEqEZofa7DQ5mOwZKFCkCN7fLKDzapuxcwtgOg+4tOae4HlxUkxMSfdBEj0TyNO3v2CE97O8hFXSulNmW9X70wBugUysQp2xu71hmStuWUDZ1AafXCbHgMlUjSNaHVNLWTyNyUODSST+AcUyROgr5nWWGNpQEgz49BeYyjr7j6ENHtUZm1RJQMX7Cduz0Nrjp+jbzgjGTUE3MY2BlToXX1amk01o8JUvc5Qgsty2AcOA74AijC0V799uTpLStX30TGldP/iCz+vTMmf3SeheD/irdOjiimf+PquT3FFjy3vivB9Q3LZkwiwsbT6hNSbbB8= ceph-817ae996-74c3-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-17T07:08:54.922 INFO:teuthology.orchestra.run.smithi179.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSfvIA4wvxb6Ujn1G3x6pvF3+OKguJQoUpNH5iCO1cqRg23SgTrSU4ZrPLDH8qFJ2y+2nFH2DBzohs1SCUcNQBQ5EQvxD9/vZW2JwiCBd4gbK/RZhXt/SB31NKN9UVJt0CVXhzDOqtA97qXdIFQZ7qxgDiCSgWqKnsYxupTTDkwo9+C8NBrcO+g+JitITFfVEqEZofa7DQ5mOwZKFCkCN7fLKDzapuxcwtgOg+4tOae4HlxUkxMSfdBEj0TyNO3v2CE97O8hFXSulNmW9X70wBugUysQp2xu71hmStuWUDZ1AafXCbHgMlUjSNaHVNLWTyNyUODSST+AcUyROgr5nWWGNpQEgz49BeYyjr7j6ENHtUZm1RJQMX7Cduz0Nrjp+jbzgjGTUE3MY2BlToXX1amk01o8JUvc5Qgsty2AcOA74AijC0V799uTpLStX30TGldP/iCz+vTMmf3SeheD/irdOjiimf+PquT3FFjy3vivB9Q3LZkwiwsbT6hNSbbB8= ceph-817ae996-74c3-11ef-bceb-c7b262605968 2024-09-17T07:08:54.935 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi179 2024-09-17T07:08:54.936 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-09-17T07:08:54.936 DEBUG:teuthology.orchestra.run.smithi179:> dd of=/etc/ceph/ceph.conf 2024-09-17T07:08:54.953 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-09-17T07:08:54.953 DEBUG:teuthology.orchestra.run.smithi179:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-17T07:08:55.010 INFO:tasks.cephadm:Adding host smithi179 to orchestrator... 2024-09-17T07:08:55.010 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph orch host add smithi179 2024-09-17T07:08:55.974 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:08:54.670430+0000 mon.a (mon.0) 89 : cluster [DBG] mgrmap e13: y(active, since 6s) 2024-09-17T07:08:57.556 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:08:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.y[23844]: Warning: Permanently added 'smithi179,172.21.15.179' (ECDSA) to the list of known hosts. 2024-09-17T07:08:57.808 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:57.224329+0000 mgr.y (mgr.14144) 5 : audit [DBG] from='client.14160 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi179", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:08:57.888 INFO:teuthology.orchestra.run.smithi044.stdout:Added host 'smithi179' 2024-09-17T07:08:58.618 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph orch host ls --format=json 2024-09-17T07:08:59.025 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:57.622988+0000 mon.a (mon.0) 90 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:08:59.025 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:57.887237+0000 mon.a (mon.0) 91 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:08:59.025 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:08:57.887614+0000 mgr.y (mgr.14144) 6 : cephadm [INF] Added host smithi179 2024-09-17T07:08:59.025 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:57.898049+0000 mon.a (mon.0) 92 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:08:59.025 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:57.899702+0000 mon.a (mon.0) 93 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:08:59.025 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:58.093243+0000 mon.a (mon.0) 94 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:08:59.025 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:08:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:08:58.279441+0000 mon.a (mon.0) 95 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:00.856 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-09-17T07:09:00.857 INFO:teuthology.orchestra.run.smithi044.stdout:[{"addr": "smithi044", "hostname": "smithi044", "labels": [], "status": ""}, {"addr": "smithi179", "hostname": "smithi179", "labels": [], "status": ""}] 2024-09-17T07:09:01.437 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-17T07:09:01.437 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph osd crush tunables default 2024-09-17T07:09:02.635 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:00.855564+0000 mgr.y (mgr.14144) 7 : audit [DBG] from='client.14162 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:09:02.635 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:01.379324+0000 mon.a (mon.0) 96 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:02.635 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:01.619488+0000 mon.a (mon.0) 97 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:02.635 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:01.622043+0000 mon.a (mon.0) 98 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:03.385 INFO:teuthology.orchestra.run.smithi044.stderr:adjusted tunables profile to default 2024-09-17T07:09:03.690 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:03 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:03.297706+0000 mon.a (mon.0) 99 : audit [INF] from='client.? 172.21.15.44:0/2649343086' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-17T07:09:03.993 INFO:tasks.cephadm:Adding mon.a on smithi044 2024-09-17T07:09:03.993 INFO:tasks.cephadm:Adding mon.c on smithi044 2024-09-17T07:09:03.993 INFO:tasks.cephadm:Adding mon.b on smithi179 2024-09-17T07:09:03.993 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph orch apply mon '3;smithi044:172.21.15.44=a;smithi044:[v2:172.21.15.44:3301,v1:172.21.15.44:6790]=c;smithi179:172.21.15.179=b' 2024-09-17T07:09:04.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:04 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:03.382362+0000 mon.a (mon.0) 100 : audit [INF] from='client.? 172.21.15.44:0/2649343086' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-17T07:09:04.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:04 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:03.382405+0000 mon.a (mon.0) 101 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-17T07:09:05.869 INFO:teuthology.orchestra.run.smithi179.stdout:Scheduled mon update... 2024-09-17T07:09:06.467 DEBUG:teuthology.orchestra.run.smithi044:mon.c> sudo journalctl -f -n 0 -u ceph-817ae996-74c3-11ef-bceb-c7b262605968@mon.c.service 2024-09-17T07:09:06.470 DEBUG:teuthology.orchestra.run.smithi179:mon.b> sudo journalctl -f -n 0 -u ceph-817ae996-74c3-11ef-bceb-c7b262605968@mon.b.service 2024-09-17T07:09:06.473 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-17T07:09:06.473 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-17T07:09:07.158 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:05.864186+0000 mgr.y (mgr.14144) 8 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi044:172.21.15.44=a;smithi044:[v2:172.21.15.44:3301,v1:172.21.15.44:6790]=c;smithi179:172.21.15.179=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:09:07.158 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:09:05.865607+0000 mgr.y (mgr.14144) 9 : cephadm [INF] Saving service mon spec with placement smithi044:172.21.15.44=a;smithi044:[v2:172.21.15.44:3301,v1:172.21.15.44:6790]=c;smithi179:172.21.15.179=b;count:3 2024-09-17T07:09:07.158 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:05.868220+0000 mon.a (mon.0) 102 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:07.158 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:05.868954+0000 mon.a (mon.0) 103 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:09:07.158 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:05.875333+0000 mon.a (mon.0) 104 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:07.158 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:05.876495+0000 mon.a (mon.0) 105 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T07:09:07.158 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:05.877299+0000 mon.a (mon.0) 106 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:09:07.158 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:09:05.878317+0000 mgr.y (mgr.14144) 10 : cephadm [INF] Deploying daemon mon.c on smithi044 2024-09-17T07:09:08.188 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-09-17T07:09:08.188 INFO:teuthology.orchestra.run.smithi179.stdout:{"epoch":1,"fsid":"817ae996-74c3-11ef-bceb-c7b262605968","modified":"2024-09-17T07:08:01.380178Z","created":"2024-09-17T07:08:01.380178Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:3300","nonce":0},{"type":"v1","addr":"172.21.15.44:6789","nonce":0}]},"addr":"172.21.15.44:6789/0","public_addr":"172.21.15.44:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-17T07:09:08.189 INFO:teuthology.orchestra.run.smithi179.stderr:dumped monmap epoch 1 2024-09-17T07:09:08.906 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:07.732641+0000 mgr.y (mgr.14144) 11 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:08.907 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:08.187357+0000 mon.a (mon.0) 107 : audit [DBG] from='client.? 172.21.15.179:0/2680447246' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T07:09:09.324 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:09 smithi044 systemd[1]: Started Ceph mon.c for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:09:09.840 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-17T07:09:09.841 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-17T07:09:14.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:09.054873+0000 mon.a (mon.0) 109 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:09:14.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:09.055034+0000 mon.a (mon.0) 110 : cluster [INF] mon.a calling monitor election 2024-09-17T07:09:14.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:09.056057+0000 mon.a (mon.0) 111 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:09:14.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:09.732876+0000 mgr.y (mgr.14144) 12 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:14.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:10.052590+0000 mon.a (mon.0) 112 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:09:14.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:11.052487+0000 mon.a (mon.0) 113 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:09:14.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:11.053930+0000 mon.c (mon.1) 1 : cluster [INF] mon.c calling monitor election 2024-09-17T07:09:14.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:11.733285+0000 mgr.y (mgr.14144) 13 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:14.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:12.052847+0000 mon.a (mon.0) 114 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:09:14.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:13.052882+0000 mon.a (mon.0) 115 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:09:14.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:14.053078+0000 mon.a (mon.0) 116 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:09:14.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:14.057790+0000 mon.a (mon.0) 117 : cluster [INF] mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-17T07:09:14.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:14.062579+0000 mon.a (mon.0) 118 : cluster [DBG] monmap e2: 2 mons at {a=[v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0],c=[v2:172.21.15.44:3301/0,v1:172.21.15.44:6790/0]} 2024-09-17T07:09:14.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:14.062622+0000 mon.a (mon.0) 119 : cluster [DBG] fsmap 2024-09-17T07:09:14.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:14.062647+0000 mon.a (mon.0) 120 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-17T07:09:14.577 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:14.062839+0000 mon.a (mon.0) 121 : cluster [DBG] mgrmap e13: y(active, since 26s) 2024-09-17T07:09:14.577 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:14.067032+0000 mon.a (mon.0) 122 : cluster [INF] overall HEALTH_OK 2024-09-17T07:09:14.577 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:14.069775+0000 mon.a (mon.0) 123 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:14.577 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:14.070584+0000 mon.a (mon.0) 124 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T07:09:14.577 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:14.071410+0000 mon.a (mon.0) 125 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:09:14.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:09.054873+0000 mon.a (mon.0) 109 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:09:14.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:09.055034+0000 mon.a (mon.0) 110 : cluster [INF] mon.a calling monitor election 2024-09-17T07:09:14.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:09.056057+0000 mon.a (mon.0) 111 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:09:14.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:09.732876+0000 mgr.y (mgr.14144) 12 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:14.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:10.052590+0000 mon.a (mon.0) 112 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:09:14.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:11.052487+0000 mon.a (mon.0) 113 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:09:14.578 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:11.053930+0000 mon.c (mon.1) 1 : cluster [INF] mon.c calling monitor election 2024-09-17T07:09:14.578 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:11.733285+0000 mgr.y (mgr.14144) 13 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:14.578 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:12.052847+0000 mon.a (mon.0) 114 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:09:14.578 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:13.052882+0000 mon.a (mon.0) 115 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:09:14.578 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:14.053078+0000 mon.a (mon.0) 116 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:09:14.578 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:14.057790+0000 mon.a (mon.0) 117 : cluster [INF] mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-17T07:09:14.578 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:14.062579+0000 mon.a (mon.0) 118 : cluster [DBG] monmap e2: 2 mons at {a=[v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0],c=[v2:172.21.15.44:3301/0,v1:172.21.15.44:6790/0]} 2024-09-17T07:09:14.578 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:14.062622+0000 mon.a (mon.0) 119 : cluster [DBG] fsmap 2024-09-17T07:09:14.578 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:14.062647+0000 mon.a (mon.0) 120 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-17T07:09:14.578 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:14.062839+0000 mon.a (mon.0) 121 : cluster [DBG] mgrmap e13: y(active, since 26s) 2024-09-17T07:09:14.578 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:14.067032+0000 mon.a (mon.0) 122 : cluster [INF] overall HEALTH_OK 2024-09-17T07:09:14.578 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:14.069775+0000 mon.a (mon.0) 123 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:14.578 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:14.070584+0000 mon.a (mon.0) 124 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T07:09:14.578 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:14.071410+0000 mon.a (mon.0) 125 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:09:14.742 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-09-17T07:09:14.742 INFO:teuthology.orchestra.run.smithi179.stdout:{"epoch":2,"fsid":"817ae996-74c3-11ef-bceb-c7b262605968","modified":"2024-09-17T07:09:09.052966Z","created":"2024-09-17T07:08:01.380178Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:3300","nonce":0},{"type":"v1","addr":"172.21.15.44:6789","nonce":0}]},"addr":"172.21.15.44:6789/0","public_addr":"172.21.15.44:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:3301","nonce":0},{"type":"v1","addr":"172.21.15.44:6790","nonce":0}]},"addr":"172.21.15.44:6790/0","public_addr":"172.21.15.44:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-17T07:09:14.743 INFO:teuthology.orchestra.run.smithi179.stderr:dumped monmap epoch 2 2024-09-17T07:09:15.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:13.733611+0000 mgr.y (mgr.14144) 14 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:15.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:09:14.072096+0000 mgr.y (mgr.14144) 15 : cephadm [INF] Deploying daemon mon.b on smithi179 2024-09-17T07:09:15.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:14.741294+0000 mon.a (mon.0) 126 : audit [DBG] from='client.? 172.21.15.179:0/3662406129' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T07:09:15.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:15.053097+0000 mon.a (mon.0) 127 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:09:15.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:13.733611+0000 mgr.y (mgr.14144) 14 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:15.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:09:14.072096+0000 mgr.y (mgr.14144) 15 : cephadm [INF] Deploying daemon mon.b on smithi179 2024-09-17T07:09:15.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:14.741294+0000 mon.a (mon.0) 126 : audit [DBG] from='client.? 172.21.15.179:0/3662406129' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T07:09:15.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:15.053097+0000 mon.a (mon.0) 127 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:09:16.610 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-17T07:09:16.611 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-17T07:09:17.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:17 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:15.733844+0000 mgr.y (mgr.14144) 16 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:17.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:17 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:15.733844+0000 mgr.y (mgr.14144) 16 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:18.924 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-09-17T07:09:18.924 INFO:teuthology.orchestra.run.smithi179.stdout:{"epoch":2,"fsid":"817ae996-74c3-11ef-bceb-c7b262605968","modified":"2024-09-17T07:09:09.052966Z","created":"2024-09-17T07:08:01.380178Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:3300","nonce":0},{"type":"v1","addr":"172.21.15.44:6789","nonce":0}]},"addr":"172.21.15.44:6789/0","public_addr":"172.21.15.44:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:3301","nonce":0},{"type":"v1","addr":"172.21.15.44:6790","nonce":0}]},"addr":"172.21.15.44:6790/0","public_addr":"172.21.15.44:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-17T07:09:18.925 INFO:teuthology.orchestra.run.smithi179.stderr:dumped monmap epoch 2 2024-09-17T07:09:19.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:17.734033+0000 mgr.y (mgr.14144) 17 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:19.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:18.923031+0000 mon.a (mon.0) 128 : audit [DBG] from='client.? 172.21.15.179:0/862438764' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T07:09:19.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:17.734033+0000 mgr.y (mgr.14144) 17 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:19.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:18.923031+0000 mon.a (mon.0) 128 : audit [DBG] from='client.? 172.21.15.179:0/862438764' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T07:09:20.780 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-17T07:09:20.781 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-17T07:09:24.783 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:24 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:24.347490+0000 mon.a (mon.0) 146 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:09:24.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:19.324705+0000 mon.a (mon.0) 130 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:09:24.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:19.324956+0000 mon.a (mon.0) 131 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:09:24.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:19.325116+0000 mon.a (mon.0) 132 : cluster [INF] mon.a calling monitor election 2024-09-17T07:09:24.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:19.326455+0000 mon.a (mon.0) 133 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:09:24.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:19.330535+0000 mon.c (mon.1) 2 : cluster [INF] mon.c calling monitor election 2024-09-17T07:09:24.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:19.734314+0000 mgr.y (mgr.14144) 18 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:24.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:20.320824+0000 mon.a (mon.0) 134 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:09:24.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:21.321109+0000 mon.a (mon.0) 135 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:09:24.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:21.321685+0000 mon.b (mon.2) 1 : cluster [INF] mon.b calling monitor election 2024-09-17T07:09:24.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:21.734606+0000 mgr.y (mgr.14144) 19 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:24.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:22.320784+0000 mon.a (mon.0) 136 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:09:24.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:23.321253+0000 mon.a (mon.0) 137 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:09:24.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:24.321101+0000 mon.a (mon.0) 138 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:09:24.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:24.328760+0000 mon.a (mon.0) 139 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T07:09:24.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:24.335708+0000 mon.a (mon.0) 140 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0],b=[v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0],c=[v2:172.21.15.44:3301/0,v1:172.21.15.44:6790/0]} 2024-09-17T07:09:24.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:24.335754+0000 mon.a (mon.0) 141 : cluster [DBG] fsmap 2024-09-17T07:09:24.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:24.335781+0000 mon.a (mon.0) 142 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-17T07:09:24.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:19.324705+0000 mon.a (mon.0) 130 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:09:24.828 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:19.324956+0000 mon.a (mon.0) 131 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:09:24.828 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:19.325116+0000 mon.a (mon.0) 132 : cluster [INF] mon.a calling monitor election 2024-09-17T07:09:24.828 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:19.326455+0000 mon.a (mon.0) 133 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:09:24.828 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:19.330535+0000 mon.c (mon.1) 2 : cluster [INF] mon.c calling monitor election 2024-09-17T07:09:24.828 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:19.734314+0000 mgr.y (mgr.14144) 18 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:24.828 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:20.320824+0000 mon.a (mon.0) 134 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:09:24.828 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:21.321109+0000 mon.a (mon.0) 135 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:09:24.828 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:21.321685+0000 mon.b (mon.2) 1 : cluster [INF] mon.b calling monitor election 2024-09-17T07:09:24.828 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:21.734606+0000 mgr.y (mgr.14144) 19 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:24.828 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:22.320784+0000 mon.a (mon.0) 136 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:09:24.828 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:23.321253+0000 mon.a (mon.0) 137 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:09:24.828 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:24.321101+0000 mon.a (mon.0) 138 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:09:24.828 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:24.328760+0000 mon.a (mon.0) 139 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T07:09:24.829 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:24.335708+0000 mon.a (mon.0) 140 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0],b=[v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0],c=[v2:172.21.15.44:3301/0,v1:172.21.15.44:6790/0]} 2024-09-17T07:09:24.829 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:24.335754+0000 mon.a (mon.0) 141 : cluster [DBG] fsmap 2024-09-17T07:09:24.829 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:24.335781+0000 mon.a (mon.0) 142 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-17T07:09:24.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:24.335982+0000 mon.a (mon.0) 143 : cluster [DBG] mgrmap e13: y(active, since 36s) 2024-09-17T07:09:24.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:24.343356+0000 mon.a (mon.0) 144 : cluster [INF] overall HEALTH_OK 2024-09-17T07:09:24.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:24.346790+0000 mon.a (mon.0) 145 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:24.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:24.347490+0000 mon.a (mon.0) 146 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:09:24.829 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:24.335982+0000 mon.a (mon.0) 143 : cluster [DBG] mgrmap e13: y(active, since 36s) 2024-09-17T07:09:24.830 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:24.343356+0000 mon.a (mon.0) 144 : cluster [INF] overall HEALTH_OK 2024-09-17T07:09:24.830 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:24.346790+0000 mon.a (mon.0) 145 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:24.830 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:24.347490+0000 mon.a (mon.0) 146 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:09:25.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:23.734891+0000 mgr.y (mgr.14144) 20 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:25.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:25.167879+0000 mon.a (mon.0) 147 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:25.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:25.321524+0000 mon.a (mon.0) 148 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:09:25.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:23.734891+0000 mgr.y (mgr.14144) 20 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:25.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:25.167879+0000 mon.a (mon.0) 147 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:25.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:25.321524+0000 mon.a (mon.0) 148 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:09:25.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:25 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:09:23.734891+0000 mgr.y (mgr.14144) 20 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:25.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:25 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:25.167879+0000 mon.a (mon.0) 147 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:25.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:25 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:25.321524+0000 mon.a (mon.0) 148 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:09:25.618 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-09-17T07:09:25.618 INFO:teuthology.orchestra.run.smithi179.stdout:{"epoch":3,"fsid":"817ae996-74c3-11ef-bceb-c7b262605968","modified":"2024-09-17T07:09:19.321159Z","created":"2024-09-17T07:08:01.380178Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:3300","nonce":0},{"type":"v1","addr":"172.21.15.44:6789","nonce":0}]},"addr":"172.21.15.44:6789/0","public_addr":"172.21.15.44:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:3301","nonce":0},{"type":"v1","addr":"172.21.15.44:6790","nonce":0}]},"addr":"172.21.15.44:6790/0","public_addr":"172.21.15.44:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:3300","nonce":0},{"type":"v1","addr":"172.21.15.179:6789","nonce":0}]},"addr":"172.21.15.179:6789/0","public_addr":"172.21.15.179:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-17T07:09:25.620 INFO:teuthology.orchestra.run.smithi179.stderr:dumped monmap epoch 3 2024-09-17T07:09:26.220 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-17T07:09:26.220 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph config generate-minimal-conf 2024-09-17T07:09:26.665 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:25.455797+0000 mon.a (mon.0) 149 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:26.665 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:25.461352+0000 mon.a (mon.0) 150 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:26.665 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:09:25.461998+0000 mgr.y (mgr.14144) 21 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2024-09-17T07:09:26.665 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:25.462180+0000 mon.a (mon.0) 151 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T07:09:26.665 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:25.462714+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T07:09:26.666 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:25.463300+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:09:26.666 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:09:25.463927+0000 mgr.y (mgr.14144) 22 : cephadm [INF] Reconfiguring daemon mon.a on smithi044 2024-09-17T07:09:26.666 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:25.618561+0000 mon.a (mon.0) 154 : audit [DBG] from='client.? 172.21.15.179:0/3634873937' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T07:09:26.666 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:25.455797+0000 mon.a (mon.0) 149 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:26.666 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:25.461352+0000 mon.a (mon.0) 150 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:26.666 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:09:25.461998+0000 mgr.y (mgr.14144) 21 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2024-09-17T07:09:26.666 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:25.462180+0000 mon.a (mon.0) 151 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T07:09:26.666 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:25.462714+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T07:09:26.666 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:25.463300+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:09:26.666 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:09:25.463927+0000 mgr.y (mgr.14144) 22 : cephadm [INF] Reconfiguring daemon mon.a on smithi044 2024-09-17T07:09:26.666 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:25.618561+0000 mon.a (mon.0) 154 : audit [DBG] from='client.? 172.21.15.179:0/3634873937' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T07:09:26.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:26 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:25.455797+0000 mon.a (mon.0) 149 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:26.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:26 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:25.461352+0000 mon.a (mon.0) 150 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:26.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:26 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:09:25.461998+0000 mgr.y (mgr.14144) 21 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2024-09-17T07:09:26.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:26 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:25.462180+0000 mon.a (mon.0) 151 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T07:09:26.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:26 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:25.462714+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T07:09:26.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:26 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:25.463300+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:09:26.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:26 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:09:25.463927+0000 mgr.y (mgr.14144) 22 : cephadm [INF] Reconfiguring daemon mon.a on smithi044 2024-09-17T07:09:26.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:26 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:25.618561+0000 mon.a (mon.0) 154 : audit [DBG] from='client.? 172.21.15.179:0/3634873937' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T07:09:27.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:25.735230+0000 mgr.y (mgr.14144) 23 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:27.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:26.712643+0000 mon.a (mon.0) 155 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:27.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:26.713318+0000 mon.a (mon.0) 156 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T07:09:27.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:26.713932+0000 mon.a (mon.0) 157 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T07:09:27.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:26.714499+0000 mon.a (mon.0) 158 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:09:27.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:25.735230+0000 mgr.y (mgr.14144) 23 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:27.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:26.712643+0000 mon.a (mon.0) 155 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:27.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:26.713318+0000 mon.a (mon.0) 156 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T07:09:27.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:26.713932+0000 mon.a (mon.0) 157 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T07:09:27.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:26.714499+0000 mon.a (mon.0) 158 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:09:27.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:27 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:09:25.735230+0000 mgr.y (mgr.14144) 23 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:27.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:27 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:26.712643+0000 mon.a (mon.0) 155 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:27.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:27 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:26.713318+0000 mon.a (mon.0) 156 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T07:09:27.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:27 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:26.713932+0000 mon.a (mon.0) 157 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T07:09:27.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:27 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:26.714499+0000 mon.a (mon.0) 158 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:09:28.558 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:28 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:09:26.713103+0000 mgr.y (mgr.14144) 24 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-09-17T07:09:28.558 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:28 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:09:26.714983+0000 mgr.y (mgr.14144) 25 : cephadm [INF] Reconfiguring daemon mon.c on smithi044 2024-09-17T07:09:28.558 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:28 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:09:26.713103+0000 mgr.y (mgr.14144) 24 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-09-17T07:09:28.558 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:28 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:09:26.714983+0000 mgr.y (mgr.14144) 25 : cephadm [INF] Reconfiguring daemon mon.c on smithi044 2024-09-17T07:09:28.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:28 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:09:26.713103+0000 mgr.y (mgr.14144) 24 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-09-17T07:09:28.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:28 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:09:26.714983+0000 mgr.y (mgr.14144) 25 : cephadm [INF] Reconfiguring daemon mon.c on smithi044 2024-09-17T07:09:28.942 INFO:teuthology.orchestra.run.smithi044.stdout:# minimal ceph.conf for 817ae996-74c3-11ef-bceb-c7b262605968 2024-09-17T07:09:28.943 INFO:teuthology.orchestra.run.smithi044.stdout:[global] 2024-09-17T07:09:28.943 INFO:teuthology.orchestra.run.smithi044.stdout: fsid = 817ae996-74c3-11ef-bceb-c7b262605968 2024-09-17T07:09:28.943 INFO:teuthology.orchestra.run.smithi044.stdout: mon_host = [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] [v2:172.21.15.44:3301/0,v1:172.21.15.44:6790/0] 2024-09-17T07:09:29.732 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:27.735449+0000 mgr.y (mgr.14144) 26 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:29.732 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:28.942551+0000 mon.a (mon.0) 159 : audit [DBG] from='client.? 172.21.15.44:0/475923687' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:09:29.732 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:29.193739+0000 mon.a (mon.0) 160 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:29.732 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:29.194400+0000 mon.a (mon.0) 161 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T07:09:29.732 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:29.194987+0000 mon.a (mon.0) 162 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T07:09:29.732 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:29.195515+0000 mon.a (mon.0) 163 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:09:29.732 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:27.735449+0000 mgr.y (mgr.14144) 26 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:29.732 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:28.942551+0000 mon.a (mon.0) 159 : audit [DBG] from='client.? 172.21.15.44:0/475923687' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:09:29.733 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:29.193739+0000 mon.a (mon.0) 160 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:29.733 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:29.194400+0000 mon.a (mon.0) 161 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T07:09:29.733 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:29.194987+0000 mon.a (mon.0) 162 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T07:09:29.733 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:29.195515+0000 mon.a (mon.0) 163 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:09:29.757 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-17T07:09:29.757 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-09-17T07:09:29.757 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/ceph/ceph.conf 2024-09-17T07:09:29.788 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-09-17T07:09:29.789 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-17T07:09:29.798 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:09:27.735449+0000 mgr.y (mgr.14144) 26 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:29.798 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:28.942551+0000 mon.a (mon.0) 159 : audit [DBG] from='client.? 172.21.15.44:0/475923687' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:09:29.798 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:29.193739+0000 mon.a (mon.0) 160 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:29.798 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:29.194400+0000 mon.a (mon.0) 161 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T07:09:29.798 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:29.194987+0000 mon.a (mon.0) 162 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T07:09:29.798 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:29.195515+0000 mon.a (mon.0) 163 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:09:29.855 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-09-17T07:09:29.855 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/ceph/ceph.conf 2024-09-17T07:09:29.883 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-09-17T07:09:29.883 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-17T07:09:29.949 INFO:tasks.cephadm:Adding mgr.y on smithi044 2024-09-17T07:09:29.949 INFO:tasks.cephadm:Adding mgr.x on smithi179 2024-09-17T07:09:29.949 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph orch apply mgr '2;smithi044=y;smithi179=x' 2024-09-17T07:09:30.711 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:30 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:09:29.194181+0000 mgr.y (mgr.14144) 27 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-09-17T07:09:30.711 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:30 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:09:29.196063+0000 mgr.y (mgr.14144) 28 : cephadm [INF] Reconfiguring daemon mon.b on smithi179 2024-09-17T07:09:30.711 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:30 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:30.454819+0000 mon.a (mon.0) 164 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:30.711 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:30 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:30.455404+0000 mon.a (mon.0) 165 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:09:30.711 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:30 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:30.458924+0000 mon.a (mon.0) 166 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:30.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:09:29.194181+0000 mgr.y (mgr.14144) 27 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-09-17T07:09:30.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:09:29.196063+0000 mgr.y (mgr.14144) 28 : cephadm [INF] Reconfiguring daemon mon.b on smithi179 2024-09-17T07:09:30.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:30.454819+0000 mon.a (mon.0) 164 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:30.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:30.455404+0000 mon.a (mon.0) 165 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:09:30.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:30.458924+0000 mon.a (mon.0) 166 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:30.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:09:29.194181+0000 mgr.y (mgr.14144) 27 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-09-17T07:09:30.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:09:29.196063+0000 mgr.y (mgr.14144) 28 : cephadm [INF] Reconfiguring daemon mon.b on smithi179 2024-09-17T07:09:30.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:30.454819+0000 mon.a (mon.0) 164 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:30.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:30.455404+0000 mon.a (mon.0) 165 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:09:30.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:30.458924+0000 mon.a (mon.0) 166 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:31.669 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:09:29.735676+0000 mgr.y (mgr.14144) 29 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:31.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:29.735676+0000 mgr.y (mgr.14144) 29 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:31.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:29.735676+0000 mgr.y (mgr.14144) 29 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:32.085 INFO:teuthology.orchestra.run.smithi179.stdout:Scheduled mgr update... 2024-09-17T07:09:32.670 DEBUG:teuthology.orchestra.run.smithi179:mgr.x> sudo journalctl -f -n 0 -u ceph-817ae996-74c3-11ef-bceb-c7b262605968@mgr.x.service 2024-09-17T07:09:32.672 INFO:tasks.cephadm:Deploying OSDs... 2024-09-17T07:09:32.672 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-09-17T07:09:32.672 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T07:09:32.693 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T07:09:32.693 DEBUG:teuthology.orchestra.run.smithi044:> stat /dev/vg_nvme/lv_1 2024-09-17T07:09:32.752 INFO:teuthology.orchestra.run.smithi044.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T07:09:32.752 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T07:09:32.752 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-17T07:09:32.752 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T07:09:32.752 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T07:09:32.753 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2024-09-17 07:09:08.546045858 +0000 2024-09-17T07:09:32.753 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2024-09-17 07:07:11.799604302 +0000 2024-09-17T07:09:32.753 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2024-09-17 07:07:11.799604302 +0000 2024-09-17T07:09:32.753 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2024-09-17 07:07:11.799604302 +0000 2024-09-17T07:09:32.753 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T07:09:32.822 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records in 2024-09-17T07:09:32.822 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records out 2024-09-17T07:09:32.822 INFO:teuthology.orchestra.run.smithi044.stderr:512 bytes copied, 0.000284596 s, 1.8 MB/s 2024-09-17T07:09:32.823 DEBUG:teuthology.orchestra.run.smithi044:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T07:09:32.882 DEBUG:teuthology.orchestra.run.smithi044:> stat /dev/vg_nvme/lv_2 2024-09-17T07:09:32.939 INFO:teuthology.orchestra.run.smithi044.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T07:09:32.940 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T07:09:32.940 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-17T07:09:32.940 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T07:09:32.940 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T07:09:32.940 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2024-09-17 07:09:08.546045858 +0000 2024-09-17T07:09:32.940 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2024-09-17 07:07:12.137599393 +0000 2024-09-17T07:09:32.940 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2024-09-17 07:07:12.137599393 +0000 2024-09-17T07:09:32.940 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2024-09-17 07:07:12.137599393 +0000 2024-09-17T07:09:32.940 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T07:09:33.006 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records in 2024-09-17T07:09:33.006 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records out 2024-09-17T07:09:33.006 INFO:teuthology.orchestra.run.smithi044.stderr:512 bytes copied, 0.000224613 s, 2.3 MB/s 2024-09-17T07:09:33.007 DEBUG:teuthology.orchestra.run.smithi044:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T07:09:33.065 DEBUG:teuthology.orchestra.run.smithi044:> stat /dev/vg_nvme/lv_3 2024-09-17T07:09:33.122 INFO:teuthology.orchestra.run.smithi044.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T07:09:33.122 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T07:09:33.122 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-17T07:09:33.122 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T07:09:33.122 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T07:09:33.122 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2024-09-17 07:09:08.546045858 +0000 2024-09-17T07:09:33.122 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2024-09-17 07:07:12.449594861 +0000 2024-09-17T07:09:33.122 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2024-09-17 07:07:12.449594861 +0000 2024-09-17T07:09:33.123 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2024-09-17 07:07:12.449594861 +0000 2024-09-17T07:09:33.123 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T07:09:33.167 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:09:31.735963+0000 mgr.y (mgr.14144) 30 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:33.167 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:32.080487+0000 mgr.y (mgr.14144) 31 : audit [DBG] from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi044=y;smithi179=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:09:33.167 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:09:32.081287+0000 mgr.y (mgr.14144) 32 : cephadm [INF] Saving service mgr spec with placement smithi044=y;smithi179=x;count:2 2024-09-17T07:09:33.167 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:32.084359+0000 mon.a (mon.0) 167 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:33.168 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:32.085085+0000 mon.a (mon.0) 168 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:09:33.168 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:32.089790+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:33.168 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:32.090668+0000 mon.a (mon.0) 170 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T07:09:33.168 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:32.095068+0000 mon.a (mon.0) 171 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-17T07:09:33.168 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:32.095895+0000 mon.a (mon.0) 172 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T07:09:33.168 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:32.096839+0000 mon.a (mon.0) 173 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:09:33.168 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:09:32.097544+0000 mgr.y (mgr.14144) 33 : cephadm [INF] Deploying daemon mgr.x on smithi179 2024-09-17T07:09:33.187 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records in 2024-09-17T07:09:33.188 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records out 2024-09-17T07:09:33.188 INFO:teuthology.orchestra.run.smithi044.stderr:512 bytes copied, 0.000236012 s, 2.2 MB/s 2024-09-17T07:09:33.189 DEBUG:teuthology.orchestra.run.smithi044:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T07:09:33.246 DEBUG:teuthology.orchestra.run.smithi044:> stat /dev/vg_nvme/lv_4 2024-09-17T07:09:33.303 INFO:teuthology.orchestra.run.smithi044.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T07:09:33.304 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T07:09:33.304 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-17T07:09:33.304 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T07:09:33.304 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T07:09:33.304 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2024-09-17 07:09:08.547045844 +0000 2024-09-17T07:09:33.304 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2024-09-17 07:07:12.785589981 +0000 2024-09-17T07:09:33.305 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2024-09-17 07:07:12.785589981 +0000 2024-09-17T07:09:33.305 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2024-09-17 07:07:12.785589981 +0000 2024-09-17T07:09:33.305 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T07:09:33.365 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:31.735963+0000 mgr.y (mgr.14144) 30 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:33.366 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:32.080487+0000 mgr.y (mgr.14144) 31 : audit [DBG] from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi044=y;smithi179=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:09:33.366 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:09:32.081287+0000 mgr.y (mgr.14144) 32 : cephadm [INF] Saving service mgr spec with placement smithi044=y;smithi179=x;count:2 2024-09-17T07:09:33.366 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:32.084359+0000 mon.a (mon.0) 167 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:33.366 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:32.085085+0000 mon.a (mon.0) 168 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:09:33.366 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:32.089790+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:33.366 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:32.090668+0000 mon.a (mon.0) 170 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T07:09:33.366 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:32.095068+0000 mon.a (mon.0) 171 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-17T07:09:33.366 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:32.095895+0000 mon.a (mon.0) 172 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T07:09:33.366 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:32.096839+0000 mon.a (mon.0) 173 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:09:33.366 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:09:32.097544+0000 mgr.y (mgr.14144) 33 : cephadm [INF] Deploying daemon mgr.x on smithi179 2024-09-17T07:09:33.367 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:31.735963+0000 mgr.y (mgr.14144) 30 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:33.367 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:32.080487+0000 mgr.y (mgr.14144) 31 : audit [DBG] from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi044=y;smithi179=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:09:33.367 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:09:32.081287+0000 mgr.y (mgr.14144) 32 : cephadm [INF] Saving service mgr spec with placement smithi044=y;smithi179=x;count:2 2024-09-17T07:09:33.367 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:32.084359+0000 mon.a (mon.0) 167 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:33.367 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:32.085085+0000 mon.a (mon.0) 168 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:09:33.367 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:32.089790+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:33.367 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:32.090668+0000 mon.a (mon.0) 170 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T07:09:33.367 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:32.095068+0000 mon.a (mon.0) 171 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-17T07:09:33.367 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:32.095895+0000 mon.a (mon.0) 172 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T07:09:33.368 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:32.096839+0000 mon.a (mon.0) 173 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:09:33.368 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:09:32.097544+0000 mgr.y (mgr.14144) 33 : cephadm [INF] Deploying daemon mgr.x on smithi179 2024-09-17T07:09:33.368 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records in 2024-09-17T07:09:33.368 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records out 2024-09-17T07:09:33.368 INFO:teuthology.orchestra.run.smithi044.stderr:512 bytes copied, 0.000171519 s, 3.0 MB/s 2024-09-17T07:09:33.369 DEBUG:teuthology.orchestra.run.smithi044:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T07:09:33.426 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-09-17T07:09:33.426 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T07:09:33.442 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T07:09:33.443 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_1 2024-09-17T07:09:33.499 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T07:09:33.499 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T07:09:33.499 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-17T07:09:33.499 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T07:09:33.499 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T07:09:33.499 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-09-17 07:09:17.932093001 +0000 2024-09-17T07:09:33.499 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-09-17 07:07:07.733943569 +0000 2024-09-17T07:09:33.499 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-09-17 07:07:07.733943569 +0000 2024-09-17T07:09:33.499 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: 2024-09-17 07:07:07.733943569 +0000 2024-09-17T07:09:33.500 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T07:09:33.564 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-09-17T07:09:33.564 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-09-17T07:09:33.565 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000189176 s, 2.7 MB/s 2024-09-17T07:09:33.566 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T07:09:33.622 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_2 2024-09-17T07:09:33.678 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T07:09:33.678 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T07:09:33.678 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-17T07:09:33.678 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T07:09:33.678 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T07:09:33.678 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-09-17 07:09:17.932093001 +0000 2024-09-17T07:09:33.679 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-09-17 07:07:08.049936238 +0000 2024-09-17T07:09:33.679 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-09-17 07:07:08.049936238 +0000 2024-09-17T07:09:33.679 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: 2024-09-17 07:07:08.049936238 +0000 2024-09-17T07:09:33.679 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T07:09:33.741 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-09-17T07:09:33.742 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-09-17T07:09:33.742 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000173208 s, 3.0 MB/s 2024-09-17T07:09:33.743 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T07:09:33.800 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_3 2024-09-17T07:09:33.856 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T07:09:33.857 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T07:09:33.857 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-17T07:09:33.857 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T07:09:33.857 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T07:09:33.857 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-09-17 07:09:17.932093001 +0000 2024-09-17T07:09:33.857 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-09-17 07:07:08.373928722 +0000 2024-09-17T07:09:33.857 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-09-17 07:07:08.373928722 +0000 2024-09-17T07:09:33.857 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: 2024-09-17 07:07:08.373928722 +0000 2024-09-17T07:09:33.857 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T07:09:33.977 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-09-17T07:09:33.977 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-09-17T07:09:33.977 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.00022313 s, 2.3 MB/s 2024-09-17T07:09:33.978 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T07:09:33.994 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_4 2024-09-17T07:09:34.053 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T07:09:34.053 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T07:09:34.053 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-17T07:09:34.053 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T07:09:34.053 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T07:09:34.054 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-09-17 07:09:33.936723878 +0000 2024-09-17T07:09:34.054 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-09-17 07:07:08.703921066 +0000 2024-09-17T07:09:34.054 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-09-17 07:07:08.703921066 +0000 2024-09-17T07:09:34.054 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: 2024-09-17 07:07:08.703921066 +0000 2024-09-17T07:09:34.054 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T07:09:34.150 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-09-17T07:09:34.150 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-09-17T07:09:34.151 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000218347 s, 2.3 MB/s 2024-09-17T07:09:34.152 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T07:09:34.167 INFO:tasks.cephadm:Deploying osd.0 on smithi044 with /dev/vg_nvme/lv_4... 2024-09-17T07:09:34.168 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-09-17T07:09:34.441 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:09:34 smithi179 podman[24825]: 2024-09-17 07:09:34.242007082 +0000 UTC m=+0.015918131 image pull 24ecd6d5f14c2268c5d2c666aa151456949cf76c6747d873c6fcf41f6f672d15 quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 2024-09-17T07:09:34.864 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:09:34 smithi179 podman[24825]: 2024-09-17 07:09:34.549047108 +0000 UTC m=+0.322958144 container init 1260ac34e181076e6ff0914c6a7020cde4d19cfbc2bb6903d4b4520f3bc76ce9 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x, GIT_BRANCH=HEAD, ceph=True, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, GIT_CLEAN=True, maintainer=Dimitri Savineau , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20201204, CEPH_POINT_RELEASE=-16.2.0) 2024-09-17T07:09:34.865 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:09:34 smithi179 podman[24825]: 2024-09-17 07:09:34.552533763 +0000 UTC m=+0.326444809 container start 1260ac34e181076e6ff0914c6a7020cde4d19cfbc2bb6903d4b4520f3bc76ce9 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20201204, CEPH_POINT_RELEASE=-16.2.0, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image, GIT_CLEAN=True, maintainer=Dimitri Savineau , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=HEAD) 2024-09-17T07:09:34.865 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:09:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[24840]: debug 2024-09-17T07:09:34.571+0000 7f1f33660700 1 -- 172.21.15.179:0/4250318598 <== mon.2 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56386f8aa4e0 con 0x56386f87f000 2024-09-17T07:09:34.865 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:09:34 smithi179 bash[24825]: 1260ac34e181076e6ff0914c6a7020cde4d19cfbc2bb6903d4b4520f3bc76ce9 2024-09-17T07:09:34.865 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:09:34 smithi179 systemd[1]: Started Ceph mgr.x for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:09:35.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:35 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:09:33.736284+0000 mgr.y (mgr.14144) 34 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:35.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:35 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:34.710810+0000 mon.a (mon.0) 174 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:35.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:35 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:34.711813+0000 mon.a (mon.0) 175 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:09:35.430 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:33.736284+0000 mgr.y (mgr.14144) 34 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:35.432 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:34.710810+0000 mon.a (mon.0) 174 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:35.432 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:34.711813+0000 mon.a (mon.0) 175 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:09:35.432 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:33.736284+0000 mgr.y (mgr.14144) 34 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:35.432 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:34.710810+0000 mon.a (mon.0) 174 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:35.432 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:34.711813+0000 mon.a (mon.0) 175 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:09:36.170 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-09-17T07:09:36.170 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-09-17T07:09:36.170 INFO:teuthology.orchestra.run.smithi044.stderr: stderr: 10+0 records in 2024-09-17T07:09:36.170 INFO:teuthology.orchestra.run.smithi044.stderr:10+0 records out 2024-09-17T07:09:36.170 INFO:teuthology.orchestra.run.smithi044.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111501 s, 940 MB/s 2024-09-17T07:09:36.170 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping successful for: 2024-09-17T07:09:36.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:35.612476+0000 mon.a (mon.0) 176 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:36.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:35.616349+0000 mon.a (mon.0) 177 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:36.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:09:35.617446+0000 mgr.y (mgr.14144) 35 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-09-17T07:09:36.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:35.617731+0000 mon.a (mon.0) 178 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T07:09:36.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:35.618673+0000 mon.a (mon.0) 179 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T07:09:36.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:35.619405+0000 mon.a (mon.0) 180 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:09:36.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:09:35.620011+0000 mgr.y (mgr.14144) 36 : cephadm [INF] Reconfiguring daemon mgr.y on smithi044 2024-09-17T07:09:36.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:35.736574+0000 mgr.y (mgr.14144) 37 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:36.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:35.612476+0000 mon.a (mon.0) 176 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:36.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:35.616349+0000 mon.a (mon.0) 177 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:36.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:09:35.617446+0000 mgr.y (mgr.14144) 35 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-09-17T07:09:36.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:35.617731+0000 mon.a (mon.0) 178 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T07:09:36.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:35.618673+0000 mon.a (mon.0) 179 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T07:09:36.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:35.619405+0000 mon.a (mon.0) 180 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:09:36.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:09:35.620011+0000 mgr.y (mgr.14144) 36 : cephadm [INF] Reconfiguring daemon mgr.y on smithi044 2024-09-17T07:09:36.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:35.736574+0000 mgr.y (mgr.14144) 37 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:36.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:35.612476+0000 mon.a (mon.0) 176 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:36.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:35.616349+0000 mon.a (mon.0) 177 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:36.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:09:35.617446+0000 mgr.y (mgr.14144) 35 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-09-17T07:09:36.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:35.617731+0000 mon.a (mon.0) 178 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T07:09:36.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:35.618673+0000 mon.a (mon.0) 179 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T07:09:36.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:35.619405+0000 mon.a (mon.0) 180 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:09:36.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:09:35.620011+0000 mgr.y (mgr.14144) 36 : cephadm [INF] Reconfiguring daemon mgr.y on smithi044 2024-09-17T07:09:36.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:09:35.736574+0000 mgr.y (mgr.14144) 37 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:37.522 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi044:vg_nvme/lv_4 2024-09-17T07:09:38.155 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:36.902095+0000 mon.a (mon.0) 181 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:38.155 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:36.902858+0000 mon.a (mon.0) 182 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:09:38.155 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:36.909822+0000 mon.a (mon.0) 183 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:38.155 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:36.902095+0000 mon.a (mon.0) 181 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:38.155 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:36.902858+0000 mon.a (mon.0) 182 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:09:38.156 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:36.909822+0000 mon.a (mon.0) 183 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:38.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:37 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:36.902095+0000 mon.a (mon.0) 181 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:38.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:37 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:36.902858+0000 mon.a (mon.0) 182 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:09:38.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:37 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:36.909822+0000 mon.a (mon.0) 183 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:39.017 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:37.736810+0000 mgr.y (mgr.14144) 38 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:39.018 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:37.736810+0000 mgr.y (mgr.14144) 38 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:39.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:38 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:09:37.736810+0000 mgr.y (mgr.14144) 38 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:40.048 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:39.452213+0000 mgr.y (mgr.14144) 39 : audit [DBG] from='client.14205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:09:40.048 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:39.452937+0000 mon.a (mon.0) 184 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:09:40.048 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:09:39.454096+0000 mgr.y (mgr.14144) 40 : cephadm [INF] Applying service osd.None on host smithi044... 2024-09-17T07:09:40.048 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:39.454316+0000 mon.a (mon.0) 185 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:09:40.048 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:39.454827+0000 mon.a (mon.0) 186 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:09:40.048 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:39.466276+0000 mon.a (mon.0) 187 : cluster [DBG] Standby manager daemon x started 2024-09-17T07:09:40.048 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:39.467274+0000 mon.b (mon.2) 2 : audit [DBG] from='mgr.? 172.21.15.179:0/3545303566' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T07:09:40.048 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:39.467559+0000 mon.b (mon.2) 3 : audit [DBG] from='mgr.? 172.21.15.179:0/3545303566' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T07:09:40.048 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:39.468315+0000 mon.b (mon.2) 4 : audit [DBG] from='mgr.? 172.21.15.179:0/3545303566' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T07:09:40.048 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:39.468572+0000 mon.b (mon.2) 5 : audit [DBG] from='mgr.? 172.21.15.179:0/3545303566' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T07:09:40.049 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:39.452213+0000 mgr.y (mgr.14144) 39 : audit [DBG] from='client.14205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:09:40.049 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:39.452937+0000 mon.a (mon.0) 184 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:09:40.049 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:09:39.454096+0000 mgr.y (mgr.14144) 40 : cephadm [INF] Applying service osd.None on host smithi044... 2024-09-17T07:09:40.049 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:39.454316+0000 mon.a (mon.0) 185 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:09:40.049 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:39.454827+0000 mon.a (mon.0) 186 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:09:40.049 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:39.466276+0000 mon.a (mon.0) 187 : cluster [DBG] Standby manager daemon x started 2024-09-17T07:09:40.049 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:39.467274+0000 mon.b (mon.2) 2 : audit [DBG] from='mgr.? 172.21.15.179:0/3545303566' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T07:09:40.049 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:39.467559+0000 mon.b (mon.2) 3 : audit [DBG] from='mgr.? 172.21.15.179:0/3545303566' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T07:09:40.049 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:39.468315+0000 mon.b (mon.2) 4 : audit [DBG] from='mgr.? 172.21.15.179:0/3545303566' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T07:09:40.049 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:39.468572+0000 mon.b (mon.2) 5 : audit [DBG] from='mgr.? 172.21.15.179:0/3545303566' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T07:09:40.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:39.452213+0000 mgr.y (mgr.14144) 39 : audit [DBG] from='client.14205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:09:40.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:39.452937+0000 mon.a (mon.0) 184 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:09:40.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:09:39.454096+0000 mgr.y (mgr.14144) 40 : cephadm [INF] Applying service osd.None on host smithi044... 2024-09-17T07:09:40.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:39.454316+0000 mon.a (mon.0) 185 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:09:40.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:39.454827+0000 mon.a (mon.0) 186 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:09:40.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:09:39.466276+0000 mon.a (mon.0) 187 : cluster [DBG] Standby manager daemon x started 2024-09-17T07:09:40.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:39.467274+0000 mon.b (mon.2) 2 : audit [DBG] from='mgr.? 172.21.15.179:0/3545303566' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T07:09:40.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:39.467559+0000 mon.b (mon.2) 3 : audit [DBG] from='mgr.? 172.21.15.179:0/3545303566' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T07:09:40.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:39.468315+0000 mon.b (mon.2) 4 : audit [DBG] from='mgr.? 172.21.15.179:0/3545303566' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T07:09:40.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:39.468572+0000 mon.b (mon.2) 5 : audit [DBG] from='mgr.? 172.21.15.179:0/3545303566' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T07:09:41.045 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:39.737169+0000 mgr.y (mgr.14144) 41 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:41.045 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:39.916236+0000 mon.a (mon.0) 188 : cluster [DBG] mgrmap e14: y(active, since 52s), standbys: x 2024-09-17T07:09:41.045 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:39.916569+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T07:09:41.045 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:39.737169+0000 mgr.y (mgr.14144) 41 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:41.045 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:39.916236+0000 mon.a (mon.0) 188 : cluster [DBG] mgrmap e14: y(active, since 52s), standbys: x 2024-09-17T07:09:41.045 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:39.916569+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T07:09:41.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:40 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:09:39.737169+0000 mgr.y (mgr.14144) 41 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:41.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:40 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:09:39.916236+0000 mon.a (mon.0) 188 : cluster [DBG] mgrmap e14: y(active, since 52s), standbys: x 2024-09-17T07:09:41.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:40 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:39.916569+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T07:09:42.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:41 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:41.679318+0000 mon.c (mon.1) 3 : audit [INF] from='client.? 172.21.15.44:0/1595433870' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f236893-2f40-4119-ab88-7359969cc30a"}]: dispatch 2024-09-17T07:09:42.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:41 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:41.679713+0000 mon.a (mon.0) 190 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f236893-2f40-4119-ab88-7359969cc30a"}]: dispatch 2024-09-17T07:09:42.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:41 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:41.682998+0000 mon.a (mon.0) 191 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7f236893-2f40-4119-ab88-7359969cc30a"}]': finished 2024-09-17T07:09:42.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:41 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:41.683060+0000 mon.a (mon.0) 192 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-09-17T07:09:42.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:41 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:41.683160+0000 mon.a (mon.0) 193 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T07:09:42.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:41 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:41.679318+0000 mon.c (mon.1) 3 : audit [INF] from='client.? 172.21.15.44:0/1595433870' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f236893-2f40-4119-ab88-7359969cc30a"}]: dispatch 2024-09-17T07:09:42.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:41 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:41.679713+0000 mon.a (mon.0) 190 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f236893-2f40-4119-ab88-7359969cc30a"}]: dispatch 2024-09-17T07:09:42.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:41 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:41.682998+0000 mon.a (mon.0) 191 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7f236893-2f40-4119-ab88-7359969cc30a"}]': finished 2024-09-17T07:09:42.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:41 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:41.683060+0000 mon.a (mon.0) 192 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-09-17T07:09:42.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:41 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:41.683160+0000 mon.a (mon.0) 193 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T07:09:42.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:41 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:41.679318+0000 mon.c (mon.1) 3 : audit [INF] from='client.? 172.21.15.44:0/1595433870' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f236893-2f40-4119-ab88-7359969cc30a"}]: dispatch 2024-09-17T07:09:42.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:41 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:41.679713+0000 mon.a (mon.0) 190 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f236893-2f40-4119-ab88-7359969cc30a"}]: dispatch 2024-09-17T07:09:42.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:41 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:41.682998+0000 mon.a (mon.0) 191 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7f236893-2f40-4119-ab88-7359969cc30a"}]': finished 2024-09-17T07:09:42.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:41 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:09:41.683060+0000 mon.a (mon.0) 192 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-09-17T07:09:42.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:41 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:41.683160+0000 mon.a (mon.0) 193 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T07:09:43.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:41.737429+0000 mgr.y (mgr.14144) 42 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:43.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:42.129519+0000 mon.a (mon.0) 194 : audit [DBG] from='client.? 172.21.15.44:0/850323189' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:09:43.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:41.737429+0000 mgr.y (mgr.14144) 42 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:43.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:42.129519+0000 mon.a (mon.0) 194 : audit [DBG] from='client.? 172.21.15.44:0/850323189' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:09:43.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:09:41.737429+0000 mgr.y (mgr.14144) 42 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:43.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:42.129519+0000 mon.a (mon.0) 194 : audit [DBG] from='client.? 172.21.15.44:0/850323189' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:09:45.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:43.737664+0000 mgr.y (mgr.14144) 43 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:45.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:43.737664+0000 mgr.y (mgr.14144) 43 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:45.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:09:43.737664+0000 mgr.y (mgr.14144) 43 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:47.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:45.737925+0000 mgr.y (mgr.14144) 44 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:47.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:45.737925+0000 mgr.y (mgr.14144) 44 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:47.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:09:45.737925+0000 mgr.y (mgr.14144) 44 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:48.150 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:47.678160+0000 mon.a (mon.0) 195 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-17T07:09:48.150 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:47.678744+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:09:48.150 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:47.863464+0000 mon.a (mon.0) 197 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:09:48.150 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:47.864580+0000 mon.a (mon.0) 198 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:09:48.150 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:47.678160+0000 mon.a (mon.0) 195 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-17T07:09:48.150 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:47.678744+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:09:48.150 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:47.863464+0000 mon.a (mon.0) 197 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:09:48.150 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:47.864580+0000 mon.a (mon.0) 198 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:09:48.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:47.678160+0000 mon.a (mon.0) 195 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-17T07:09:48.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:47.678744+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:09:48.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:47.863464+0000 mon.a (mon.0) 197 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:09:48.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:47.864580+0000 mon.a (mon.0) 198 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:09:49.284 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:09:47.679252+0000 mgr.y (mgr.14144) 45 : cephadm [INF] Deploying daemon osd.0 on smithi044 2024-09-17T07:09:49.284 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:47.738110+0000 mgr.y (mgr.14144) 46 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:49.284 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:09:47.679252+0000 mgr.y (mgr.14144) 45 : cephadm [INF] Deploying daemon osd.0 on smithi044 2024-09-17T07:09:49.284 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:47.738110+0000 mgr.y (mgr.14144) 46 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:49.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:09:47.679252+0000 mgr.y (mgr.14144) 45 : cephadm [INF] Deploying daemon osd.0 on smithi044 2024-09-17T07:09:49.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:09:47.738110+0000 mgr.y (mgr.14144) 46 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:50.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:09:49.738324+0000 mgr.y (mgr.14144) 47 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:50.871 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:49.738324+0000 mgr.y (mgr.14144) 47 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:50.871 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:49.738324+0000 mgr.y (mgr.14144) 47 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:51.131 INFO:teuthology.orchestra.run.smithi044.stdout:Created osd(s) 0 on host 'smithi044' 2024-09-17T07:09:51.809 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:51.121397+0000 mon.a (mon.0) 199 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:09:51.809 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:51.124896+0000 mon.a (mon.0) 200 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:51.810 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:51.129371+0000 mon.a (mon.0) 201 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:51.810 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:51.121397+0000 mon.a (mon.0) 199 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:09:51.810 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:51.124896+0000 mon.a (mon.0) 200 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:51.810 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:51.129371+0000 mon.a (mon.0) 201 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:51.834 DEBUG:teuthology.orchestra.run.smithi044:osd.0> sudo journalctl -f -n 0 -u ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.0.service 2024-09-17T07:09:51.838 INFO:tasks.cephadm:Deploying osd.1 on smithi044 with /dev/vg_nvme/lv_3... 2024-09-17T07:09:51.838 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-09-17T07:09:51.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:51.121397+0000 mon.a (mon.0) 199 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:09:51.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:51.124896+0000 mon.a (mon.0) 200 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:51.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:51.129371+0000 mon.a (mon.0) 201 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:52.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:51.738673+0000 mgr.y (mgr.14144) 48 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:52.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:51.738673+0000 mgr.y (mgr.14144) 48 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:53.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:09:51.738673+0000 mgr.y (mgr.14144) 48 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:54.384 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-09-17T07:09:54.384 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-09-17T07:09:54.384 INFO:teuthology.orchestra.run.smithi044.stderr: stderr: 10+0 records in 2024-09-17T07:09:54.385 INFO:teuthology.orchestra.run.smithi044.stderr:10+0 records out 2024-09-17T07:09:54.385 INFO:teuthology.orchestra.run.smithi044.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0127324 s, 824 MB/s 2024-09-17T07:09:54.385 INFO:teuthology.orchestra.run.smithi044.stderr: stderr: 2024-09-17T07:09:54.385 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping successful for: 2024-09-17T07:09:54.487 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:09:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.0[29263]: debug 2024-09-17T07:09:54.218+0000 7f6d9ca94080 -1 osd.0 0 log_to_monitors {default=true} 2024-09-17T07:09:55.035 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi044:vg_nvme/lv_3 2024-09-17T07:09:55.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:53.738925+0000 mgr.y (mgr.14144) 49 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:55.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:54.081622+0000 mon.a (mon.0) 202 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:55.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:54.223170+0000 mon.c (mon.1) 4 : audit [INF] from='osd.0 [v2:172.21.15.44:6802/1979937622,v1:172.21.15.44:6803/1979937622]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T07:09:55.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:54.223517+0000 mon.a (mon.0) 203 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T07:09:55.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:53.738925+0000 mgr.y (mgr.14144) 49 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:55.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:54.081622+0000 mon.a (mon.0) 202 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:55.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:54.223170+0000 mon.c (mon.1) 4 : audit [INF] from='osd.0 [v2:172.21.15.44:6802/1979937622,v1:172.21.15.44:6803/1979937622]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T07:09:55.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:54.223517+0000 mon.a (mon.0) 203 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T07:09:55.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:09:53.738925+0000 mgr.y (mgr.14144) 49 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:55.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:54.081622+0000 mon.a (mon.0) 202 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:55.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:54.223170+0000 mon.c (mon.1) 4 : audit [INF] from='osd.0 [v2:172.21.15.44:6802/1979937622,v1:172.21.15.44:6803/1979937622]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T07:09:55.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:54.223517+0000 mon.a (mon.0) 203 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T07:09:56.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:56 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:55.085559+0000 mon.a (mon.0) 204 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-17T07:09:56.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:56 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:09:55.085653+0000 mon.a (mon.0) 205 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-09-17T07:09:56.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:56 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:55.086078+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T07:09:56.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:56 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:55.086434+0000 mon.c (mon.1) 5 : audit [INF] from='osd.0 [v2:172.21.15.44:6802/1979937622,v1:172.21.15.44:6803/1979937622]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-09-17T07:09:56.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:56 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:55.090739+0000 mon.a (mon.0) 207 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-09-17T07:09:56.374 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:09:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.0[29263]: debug 2024-09-17T07:09:56.095+0000 7f6d93005700 -1 osd.0 0 waiting for initial osdmap 2024-09-17T07:09:56.374 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:09:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.0[29263]: debug 2024-09-17T07:09:56.098+0000 7f6d8f781700 -1 osd.0 7 set_numa_affinity unable to identify public interface 'ens1f0' numa node: (0) Success 2024-09-17T07:09:56.374 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:55.085559+0000 mon.a (mon.0) 204 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-17T07:09:56.374 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:55.085653+0000 mon.a (mon.0) 205 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-09-17T07:09:56.374 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:55.086078+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T07:09:56.374 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:55.086434+0000 mon.c (mon.1) 5 : audit [INF] from='osd.0 [v2:172.21.15.44:6802/1979937622,v1:172.21.15.44:6803/1979937622]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-09-17T07:09:56.374 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:55.090739+0000 mon.a (mon.0) 207 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-09-17T07:09:56.375 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:55.085559+0000 mon.a (mon.0) 204 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-17T07:09:56.375 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:55.085653+0000 mon.a (mon.0) 205 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-09-17T07:09:56.375 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:55.086078+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T07:09:56.375 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:55.086434+0000 mon.c (mon.1) 5 : audit [INF] from='osd.0 [v2:172.21.15.44:6802/1979937622,v1:172.21.15.44:6803/1979937622]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-09-17T07:09:56.375 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:55.090739+0000 mon.a (mon.0) 207 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-09-17T07:09:57.324 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:55.739262+0000 mgr.y (mgr.14144) 50 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:57.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:56.091462+0000 mon.a (mon.0) 208 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-09-17T07:09:57.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:56.091514+0000 mon.a (mon.0) 209 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-09-17T07:09:57.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:56.091916+0000 mon.a (mon.0) 210 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T07:09:57.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:56.096186+0000 mon.a (mon.0) 211 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T07:09:57.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:55.739262+0000 mgr.y (mgr.14144) 50 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:57.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:56.091462+0000 mon.a (mon.0) 208 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-09-17T07:09:57.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:56.091514+0000 mon.a (mon.0) 209 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-09-17T07:09:57.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:56.091916+0000 mon.a (mon.0) 210 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T07:09:57.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:56.096186+0000 mon.a (mon.0) 211 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T07:09:57.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:09:55.739262+0000 mgr.y (mgr.14144) 50 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:57.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:56.091462+0000 mon.a (mon.0) 208 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-09-17T07:09:57.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:09:56.091514+0000 mon.a (mon.0) 209 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-09-17T07:09:57.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:56.091916+0000 mon.a (mon.0) 210 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T07:09:57.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:56.096186+0000 mon.a (mon.0) 211 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T07:09:58.214 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:55.262377+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T07:09:58.215 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:55.262569+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T07:09:58.215 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:57.096577+0000 mon.a (mon.0) 212 : cluster [INF] osd.0 [v2:172.21.15.44:6802/1979937622,v1:172.21.15.44:6803/1979937622] boot 2024-09-17T07:09:58.215 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:57.096613+0000 mon.a (mon.0) 213 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-09-17T07:09:58.215 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:57.096950+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T07:09:58.215 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:58.045218+0000 mon.a (mon.0) 215 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:09:58.215 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:58.046559+0000 mon.a (mon.0) 216 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:09:58.215 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:58.047109+0000 mon.a (mon.0) 217 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:09:58.215 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:55.262377+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T07:09:58.215 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:55.262569+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T07:09:58.215 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:57.096577+0000 mon.a (mon.0) 212 : cluster [INF] osd.0 [v2:172.21.15.44:6802/1979937622,v1:172.21.15.44:6803/1979937622] boot 2024-09-17T07:09:58.215 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:57.096613+0000 mon.a (mon.0) 213 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-09-17T07:09:58.216 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:57.096950+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T07:09:58.216 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:58.045218+0000 mon.a (mon.0) 215 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:09:58.216 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:58.046559+0000 mon.a (mon.0) 216 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:09:58.216 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:58.047109+0000 mon.a (mon.0) 217 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:09:58.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:58 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:09:55.262377+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T07:09:58.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:58 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:09:55.262569+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T07:09:58.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:58 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:09:57.096577+0000 mon.a (mon.0) 212 : cluster [INF] osd.0 [v2:172.21.15.44:6802/1979937622,v1:172.21.15.44:6803/1979937622] boot 2024-09-17T07:09:58.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:58 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:09:57.096613+0000 mon.a (mon.0) 213 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-09-17T07:09:58.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:58 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:57.096950+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T07:09:58.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:58 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:58.045218+0000 mon.a (mon.0) 215 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:09:58.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:58 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:58.046559+0000 mon.a (mon.0) 216 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:09:58.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:58 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:58.047109+0000 mon.a (mon.0) 217 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:09:59.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:59 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:09:57.739543+0000 mgr.y (mgr.14144) 51 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:59.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:59 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:58.044446+0000 mgr.y (mgr.14144) 52 : audit [DBG] from='client.24142 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:09:59.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:59 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:09:58.046346+0000 mgr.y (mgr.14144) 53 : cephadm [INF] Applying service osd.None on host smithi044... 2024-09-17T07:09:59.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:59 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:09:58.104371+0000 mon.a (mon.0) 218 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-09-17T07:09:59.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:59 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:58.719830+0000 mon.a (mon.0) 219 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:59.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:59 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:58.941880+0000 mon.a (mon.0) 220 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:59.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:59 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:58.942997+0000 mon.a (mon.0) 221 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:09:59.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:09:59 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:09:58.946867+0000 mon.a (mon.0) 222 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:59.531 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:59 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:57.739543+0000 mgr.y (mgr.14144) 51 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:59.532 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:59 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:58.044446+0000 mgr.y (mgr.14144) 52 : audit [DBG] from='client.24142 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:09:59.532 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:59 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:09:58.046346+0000 mgr.y (mgr.14144) 53 : cephadm [INF] Applying service osd.None on host smithi044... 2024-09-17T07:09:59.532 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:59 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:58.104371+0000 mon.a (mon.0) 218 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-09-17T07:09:59.532 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:59 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:58.719830+0000 mon.a (mon.0) 219 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:59.532 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:59 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:58.941880+0000 mon.a (mon.0) 220 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:59.532 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:59 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:58.942997+0000 mon.a (mon.0) 221 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:09:59.532 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:09:59 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:09:58.946867+0000 mon.a (mon.0) 222 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:59.532 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:59 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:57.739543+0000 mgr.y (mgr.14144) 51 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T07:09:59.532 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:59 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:58.044446+0000 mgr.y (mgr.14144) 52 : audit [DBG] from='client.24142 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:09:59.533 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:59 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:09:58.046346+0000 mgr.y (mgr.14144) 53 : cephadm [INF] Applying service osd.None on host smithi044... 2024-09-17T07:09:59.533 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:59 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:58.104371+0000 mon.a (mon.0) 218 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-09-17T07:09:59.533 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:59 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:58.719830+0000 mon.a (mon.0) 219 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:59.533 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:59 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:58.941880+0000 mon.a (mon.0) 220 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:09:59.533 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:59 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:58.942997+0000 mon.a (mon.0) 221 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:09:59.533 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:09:59 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:09:58.946867+0000 mon.a (mon.0) 222 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:00.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:00 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:09:58.715795+0000 mgr.y (mgr.14144) 54 : cephadm [INF] Detected new or changed devices on smithi044 2024-09-17T07:10:00.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:00 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:00.000120+0000 mon.a (mon.0) 223 : cluster [INF] overall HEALTH_OK 2024-09-17T07:10:00.464 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:09:58.715795+0000 mgr.y (mgr.14144) 54 : cephadm [INF] Detected new or changed devices on smithi044 2024-09-17T07:10:00.464 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:00.000120+0000 mon.a (mon.0) 223 : cluster [INF] overall HEALTH_OK 2024-09-17T07:10:00.465 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:09:58.715795+0000 mgr.y (mgr.14144) 54 : cephadm [INF] Detected new or changed devices on smithi044 2024-09-17T07:10:00.465 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:00.000120+0000 mon.a (mon.0) 223 : cluster [INF] overall HEALTH_OK 2024-09-17T07:10:01.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:01 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:09:59.739941+0000 mgr.y (mgr.14144) 55 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T07:10:01.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:01 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:00.568443+0000 mon.a (mon.0) 224 : audit [INF] from='client.? 172.21.15.44:0/3500379824' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f8da14c0-ab31-42e2-be11-bd734d990923"}]: dispatch 2024-09-17T07:10:01.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:01 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:00.573218+0000 mon.a (mon.0) 225 : audit [INF] from='client.? 172.21.15.44:0/3500379824' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f8da14c0-ab31-42e2-be11-bd734d990923"}]': finished 2024-09-17T07:10:01.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:01 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:00.573292+0000 mon.a (mon.0) 226 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-09-17T07:10:01.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:01 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:00.573422+0000 mon.a (mon.0) 227 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T07:10:01.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:01 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:01.006933+0000 mon.a (mon.0) 228 : audit [DBG] from='client.? 172.21.15.44:0/1418393817' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:10:01.456 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:09:59.739941+0000 mgr.y (mgr.14144) 55 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T07:10:01.457 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:00.568443+0000 mon.a (mon.0) 224 : audit [INF] from='client.? 172.21.15.44:0/3500379824' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f8da14c0-ab31-42e2-be11-bd734d990923"}]: dispatch 2024-09-17T07:10:01.457 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:00.573218+0000 mon.a (mon.0) 225 : audit [INF] from='client.? 172.21.15.44:0/3500379824' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f8da14c0-ab31-42e2-be11-bd734d990923"}]': finished 2024-09-17T07:10:01.457 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:00.573292+0000 mon.a (mon.0) 226 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-09-17T07:10:01.457 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:00.573422+0000 mon.a (mon.0) 227 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T07:10:01.457 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:01.006933+0000 mon.a (mon.0) 228 : audit [DBG] from='client.? 172.21.15.44:0/1418393817' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:10:01.457 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:09:59.739941+0000 mgr.y (mgr.14144) 55 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T07:10:01.457 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:00.568443+0000 mon.a (mon.0) 224 : audit [INF] from='client.? 172.21.15.44:0/3500379824' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f8da14c0-ab31-42e2-be11-bd734d990923"}]: dispatch 2024-09-17T07:10:01.457 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:00.573218+0000 mon.a (mon.0) 225 : audit [INF] from='client.? 172.21.15.44:0/3500379824' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f8da14c0-ab31-42e2-be11-bd734d990923"}]': finished 2024-09-17T07:10:01.457 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:00.573292+0000 mon.a (mon.0) 226 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-09-17T07:10:01.458 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:00.573422+0000 mon.a (mon.0) 227 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T07:10:01.458 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:01.006933+0000 mon.a (mon.0) 228 : audit [DBG] from='client.? 172.21.15.44:0/1418393817' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:10:03.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:03 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:01.740260+0000 mgr.y (mgr.14144) 56 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T07:10:03.563 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:03 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:01.740260+0000 mgr.y (mgr.14144) 56 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T07:10:03.563 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:03 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:01.740260+0000 mgr.y (mgr.14144) 56 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T07:10:05.368 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:03.740616+0000 mgr.y (mgr.14144) 57 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T07:10:05.368 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:03.740616+0000 mgr.y (mgr.14144) 57 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T07:10:05.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:05 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:03.740616+0000 mgr.y (mgr.14144) 57 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T07:10:07.292 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:05.741003+0000 mgr.y (mgr.14144) 58 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T07:10:07.292 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:06.318625+0000 mon.a (mon.0) 229 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-17T07:10:07.292 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:06.319210+0000 mon.a (mon.0) 230 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:10:07.293 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:10:06.319669+0000 mgr.y (mgr.14144) 59 : cephadm [INF] Deploying daemon osd.1 on smithi044 2024-09-17T07:10:07.293 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:05.741003+0000 mgr.y (mgr.14144) 58 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T07:10:07.293 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:06.318625+0000 mon.a (mon.0) 229 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-17T07:10:07.293 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:06.319210+0000 mon.a (mon.0) 230 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:10:07.293 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:10:06.319669+0000 mgr.y (mgr.14144) 59 : cephadm [INF] Deploying daemon osd.1 on smithi044 2024-09-17T07:10:07.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:05.741003+0000 mgr.y (mgr.14144) 58 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T07:10:07.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:06.318625+0000 mon.a (mon.0) 229 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-17T07:10:07.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:06.319210+0000 mon.a (mon.0) 230 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:10:07.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:10:06.319669+0000 mgr.y (mgr.14144) 59 : cephadm [INF] Deploying daemon osd.1 on smithi044 2024-09-17T07:10:09.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:07.741294+0000 mgr.y (mgr.14144) 60 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T07:10:09.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:07.741294+0000 mgr.y (mgr.14144) 60 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T07:10:09.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:07.741294+0000 mgr.y (mgr.14144) 60 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T07:10:09.858 INFO:teuthology.orchestra.run.smithi044.stdout:Created osd(s) 1 on host 'smithi044' 2024-09-17T07:10:10.489 DEBUG:teuthology.orchestra.run.smithi044:osd.1> sudo journalctl -f -n 0 -u ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.1.service 2024-09-17T07:10:10.491 INFO:tasks.cephadm:Deploying osd.2 on smithi044 with /dev/vg_nvme/lv_2... 2024-09-17T07:10:10.492 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-09-17T07:10:10.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:09.741595+0000 mgr.y (mgr.14144) 61 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T07:10:10.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:09.844178+0000 mon.a (mon.0) 231 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:10:10.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:09.849497+0000 mon.a (mon.0) 232 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:10.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:09.856061+0000 mon.a (mon.0) 233 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:10.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:09.741595+0000 mgr.y (mgr.14144) 61 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T07:10:10.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:09.844178+0000 mon.a (mon.0) 231 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:10:10.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:09.849497+0000 mon.a (mon.0) 232 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:10.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:09.856061+0000 mon.a (mon.0) 233 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:10.953 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:09.741595+0000 mgr.y (mgr.14144) 61 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T07:10:10.953 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:09.844178+0000 mon.a (mon.0) 231 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:10:10.954 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:09.849497+0000 mon.a (mon.0) 232 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:10.954 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:09.856061+0000 mon.a (mon.0) 233 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:12.796 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:11.741952+0000 mgr.y (mgr.14144) 62 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T07:10:12.797 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:11.741952+0000 mgr.y (mgr.14144) 62 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T07:10:13.073 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-09-17T07:10:13.073 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-09-17T07:10:13.073 INFO:teuthology.orchestra.run.smithi044.stderr: stderr: 10+0 records in 2024-09-17T07:10:13.073 INFO:teuthology.orchestra.run.smithi044.stderr:10+0 records out 2024-09-17T07:10:13.073 INFO:teuthology.orchestra.run.smithi044.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122971 s, 853 MB/s 2024-09-17T07:10:13.073 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping successful for: 2024-09-17T07:10:13.074 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:10:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.1[31665]: debug 2024-09-17T07:10:12.967+0000 7f956ad00080 -1 osd.1 0 log_to_monitors {default=true} 2024-09-17T07:10:13.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:12 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:11.741952+0000 mgr.y (mgr.14144) 62 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T07:10:13.959 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi044:vg_nvme/lv_2 2024-09-17T07:10:14.093 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:12.827705+0000 mon.a (mon.0) 234 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:14.093 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:12.971522+0000 mon.c (mon.1) 6 : audit [INF] from='osd.1 [v2:172.21.15.44:6810/1594621847,v1:172.21.15.44:6811/1594621847]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T07:10:14.093 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:12.971892+0000 mon.a (mon.0) 235 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T07:10:14.094 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:12.827705+0000 mon.a (mon.0) 234 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:14.094 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:12.971522+0000 mon.c (mon.1) 6 : audit [INF] from='osd.1 [v2:172.21.15.44:6810/1594621847,v1:172.21.15.44:6811/1594621847]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T07:10:14.094 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:12.971892+0000 mon.a (mon.0) 235 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T07:10:14.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:13 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:12.827705+0000 mon.a (mon.0) 234 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:14.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:13 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:12.971522+0000 mon.c (mon.1) 6 : audit [INF] from='osd.1 [v2:172.21.15.44:6810/1594621847,v1:172.21.15.44:6811/1594621847]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T07:10:14.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:13 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:12.971892+0000 mon.a (mon.0) 235 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T07:10:15.074 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:10:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.1[31665]: debug 2024-09-17T07:10:14.840+0000 7f9561271700 -1 osd.1 0 waiting for initial osdmap 2024-09-17T07:10:15.074 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:10:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.1[31665]: debug 2024-09-17T07:10:14.843+0000 7f95599e5700 -1 osd.1 12 set_numa_affinity unable to identify public interface 'ens1f0' numa node: (0) Success 2024-09-17T07:10:15.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:13.742194+0000 mgr.y (mgr.14144) 63 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T07:10:15.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:13.833226+0000 mon.a (mon.0) 236 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-17T07:10:15.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:13.833312+0000 mon.a (mon.0) 237 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-09-17T07:10:15.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:13.833406+0000 mon.a (mon.0) 238 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T07:10:15.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:13.833892+0000 mon.c (mon.1) 7 : audit [INF] from='osd.1 [v2:172.21.15.44:6810/1594621847,v1:172.21.15.44:6811/1594621847]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-09-17T07:10:15.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:13.834220+0000 mon.a (mon.0) 239 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-09-17T07:10:15.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:13.742194+0000 mgr.y (mgr.14144) 63 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T07:10:15.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:13.833226+0000 mon.a (mon.0) 236 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-17T07:10:15.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:13.833312+0000 mon.a (mon.0) 237 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-09-17T07:10:15.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:13.833406+0000 mon.a (mon.0) 238 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T07:10:15.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:13.833892+0000 mon.c (mon.1) 7 : audit [INF] from='osd.1 [v2:172.21.15.44:6810/1594621847,v1:172.21.15.44:6811/1594621847]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-09-17T07:10:15.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:13.834220+0000 mon.a (mon.0) 239 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-09-17T07:10:15.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:14 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:13.742194+0000 mgr.y (mgr.14144) 63 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T07:10:15.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:14 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:13.833226+0000 mon.a (mon.0) 236 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-17T07:10:15.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:14 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:13.833312+0000 mon.a (mon.0) 237 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-09-17T07:10:15.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:14 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:13.833406+0000 mon.a (mon.0) 238 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T07:10:15.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:14 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:13.833892+0000 mon.c (mon.1) 7 : audit [INF] from='osd.1 [v2:172.21.15.44:6810/1594621847,v1:172.21.15.44:6811/1594621847]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-09-17T07:10:15.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:14 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:13.834220+0000 mon.a (mon.0) 239 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-09-17T07:10:15.909 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:14.835356+0000 mon.a (mon.0) 240 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-09-17T07:10:15.909 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:14.835413+0000 mon.a (mon.0) 241 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-09-17T07:10:15.910 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:14.836270+0000 mon.a (mon.0) 242 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T07:10:15.910 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:14.841987+0000 mon.a (mon.0) 243 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T07:10:15.910 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:14.835356+0000 mon.a (mon.0) 240 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-09-17T07:10:15.910 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:14.835413+0000 mon.a (mon.0) 241 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-09-17T07:10:15.910 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:14.836270+0000 mon.a (mon.0) 242 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T07:10:15.910 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:14.841987+0000 mon.a (mon.0) 243 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T07:10:16.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:14.835356+0000 mon.a (mon.0) 240 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-09-17T07:10:16.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:14.835413+0000 mon.a (mon.0) 241 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-09-17T07:10:16.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:14.836270+0000 mon.a (mon.0) 242 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T07:10:16.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:14.841987+0000 mon.a (mon.0) 243 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T07:10:17.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:13.977050+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T07:10:17.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:13.977149+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T07:10:17.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:15.742484+0000 mgr.y (mgr.14144) 64 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T07:10:17.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:15.842353+0000 mon.a (mon.0) 244 : cluster [INF] osd.1 [v2:172.21.15.44:6810/1594621847,v1:172.21.15.44:6811/1594621847] boot 2024-09-17T07:10:17.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:15.842393+0000 mon.a (mon.0) 245 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-09-17T07:10:17.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:15.842751+0000 mon.a (mon.0) 246 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T07:10:17.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:16.492609+0000 mgr.y (mgr.14144) 65 : audit [DBG] from='client.14259 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:10:17.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:16.493354+0000 mon.a (mon.0) 247 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:10:17.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:10:16.494400+0000 mgr.y (mgr.14144) 66 : cephadm [INF] Applying service osd.None on host smithi044... 2024-09-17T07:10:17.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:16.494602+0000 mon.a (mon.0) 248 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:10:17.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:16.495141+0000 mon.a (mon.0) 249 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:10:17.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:13.977050+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T07:10:17.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:13.977149+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T07:10:17.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:15.742484+0000 mgr.y (mgr.14144) 64 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T07:10:17.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:15.842353+0000 mon.a (mon.0) 244 : cluster [INF] osd.1 [v2:172.21.15.44:6810/1594621847,v1:172.21.15.44:6811/1594621847] boot 2024-09-17T07:10:17.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:15.842393+0000 mon.a (mon.0) 245 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-09-17T07:10:17.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:15.842751+0000 mon.a (mon.0) 246 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T07:10:17.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:16.492609+0000 mgr.y (mgr.14144) 65 : audit [DBG] from='client.14259 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:10:17.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:16.493354+0000 mon.a (mon.0) 247 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:10:17.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:10:16.494400+0000 mgr.y (mgr.14144) 66 : cephadm [INF] Applying service osd.None on host smithi044... 2024-09-17T07:10:17.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:16.494602+0000 mon.a (mon.0) 248 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:10:17.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:16.495141+0000 mon.a (mon.0) 249 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:10:17.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:16 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:13.977050+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T07:10:17.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:16 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:13.977149+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T07:10:17.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:16 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:15.742484+0000 mgr.y (mgr.14144) 64 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T07:10:17.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:16 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:15.842353+0000 mon.a (mon.0) 244 : cluster [INF] osd.1 [v2:172.21.15.44:6810/1594621847,v1:172.21.15.44:6811/1594621847] boot 2024-09-17T07:10:17.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:16 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:15.842393+0000 mon.a (mon.0) 245 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-09-17T07:10:17.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:16 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:15.842751+0000 mon.a (mon.0) 246 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T07:10:17.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:16 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:16.492609+0000 mgr.y (mgr.14144) 65 : audit [DBG] from='client.14259 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:10:17.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:16 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:16.493354+0000 mon.a (mon.0) 247 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:10:17.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:16 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:10:16.494400+0000 mgr.y (mgr.14144) 66 : cephadm [INF] Applying service osd.None on host smithi044... 2024-09-17T07:10:17.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:16 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:16.494602+0000 mon.a (mon.0) 248 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:10:17.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:16 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:16.495141+0000 mon.a (mon.0) 249 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:10:18.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:16.849127+0000 mon.a (mon.0) 250 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-09-17T07:10:18.253 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:17 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:16.849127+0000 mon.a (mon.0) 250 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-09-17T07:10:18.253 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:17 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:16.849127+0000 mon.a (mon.0) 250 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-09-17T07:10:19.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:17.742897+0000 mgr.y (mgr.14144) 67 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:10:19.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:18.723098+0000 mon.a (mon.0) 251 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:19.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:18.726779+0000 mon.a (mon.0) 252 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:19.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:18.728028+0000 mon.a (mon.0) 253 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:10:19.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:18.731997+0000 mon.a (mon.0) 254 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:19.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:17.742897+0000 mgr.y (mgr.14144) 67 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:10:19.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:18.723098+0000 mon.a (mon.0) 251 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:19.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:18.726779+0000 mon.a (mon.0) 252 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:19.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:18.728028+0000 mon.a (mon.0) 253 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:10:19.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:18.731997+0000 mon.a (mon.0) 254 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:19.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:18 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:17.742897+0000 mgr.y (mgr.14144) 67 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:10:19.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:18 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:18.723098+0000 mon.a (mon.0) 251 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:19.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:18 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:18.726779+0000 mon.a (mon.0) 252 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:19.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:18 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:18.728028+0000 mon.a (mon.0) 253 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:10:19.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:18 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:18.731997+0000 mon.a (mon.0) 254 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:20.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:10:18.718975+0000 mgr.y (mgr.14144) 68 : cephadm [INF] Detected new or changed devices on smithi044 2024-09-17T07:10:20.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:19.542298+0000 mon.a (mon.0) 255 : audit [INF] from='client.? 172.21.15.44:0/2717641058' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "68dff20f-443b-4ed3-86a5-dc84cb5b9da7"}]: dispatch 2024-09-17T07:10:20.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:19.545779+0000 mon.a (mon.0) 256 : audit [INF] from='client.? 172.21.15.44:0/2717641058' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "68dff20f-443b-4ed3-86a5-dc84cb5b9da7"}]': finished 2024-09-17T07:10:20.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:19.545830+0000 mon.a (mon.0) 257 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2024-09-17T07:10:20.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:19.545926+0000 mon.a (mon.0) 258 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:10:20.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:10:18.718975+0000 mgr.y (mgr.14144) 68 : cephadm [INF] Detected new or changed devices on smithi044 2024-09-17T07:10:20.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:19.542298+0000 mon.a (mon.0) 255 : audit [INF] from='client.? 172.21.15.44:0/2717641058' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "68dff20f-443b-4ed3-86a5-dc84cb5b9da7"}]: dispatch 2024-09-17T07:10:20.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:19.545779+0000 mon.a (mon.0) 256 : audit [INF] from='client.? 172.21.15.44:0/2717641058' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "68dff20f-443b-4ed3-86a5-dc84cb5b9da7"}]': finished 2024-09-17T07:10:20.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:19.545830+0000 mon.a (mon.0) 257 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2024-09-17T07:10:20.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:19.545926+0000 mon.a (mon.0) 258 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:10:20.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:19 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:10:18.718975+0000 mgr.y (mgr.14144) 68 : cephadm [INF] Detected new or changed devices on smithi044 2024-09-17T07:10:20.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:19 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:19.542298+0000 mon.a (mon.0) 255 : audit [INF] from='client.? 172.21.15.44:0/2717641058' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "68dff20f-443b-4ed3-86a5-dc84cb5b9da7"}]: dispatch 2024-09-17T07:10:20.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:19 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:19.545779+0000 mon.a (mon.0) 256 : audit [INF] from='client.? 172.21.15.44:0/2717641058' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "68dff20f-443b-4ed3-86a5-dc84cb5b9da7"}]': finished 2024-09-17T07:10:20.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:19 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:19.545830+0000 mon.a (mon.0) 257 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2024-09-17T07:10:20.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:19 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:19.545926+0000 mon.a (mon.0) 258 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:10:21.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:19.743195+0000 mgr.y (mgr.14144) 69 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:10:21.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:19.970929+0000 mon.a (mon.0) 259 : audit [DBG] from='client.? 172.21.15.44:0/1285994419' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:10:21.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:19.743195+0000 mgr.y (mgr.14144) 69 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:10:21.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:19.970929+0000 mon.a (mon.0) 259 : audit [DBG] from='client.? 172.21.15.44:0/1285994419' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:10:21.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:19.743195+0000 mgr.y (mgr.14144) 69 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:10:21.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:19.970929+0000 mon.a (mon.0) 259 : audit [DBG] from='client.? 172.21.15.44:0/1285994419' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:10:23.188 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:22 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:21.743581+0000 mgr.y (mgr.14144) 70 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:10:23.188 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:22 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:21.743581+0000 mgr.y (mgr.14144) 70 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:10:23.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:21.743581+0000 mgr.y (mgr.14144) 70 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:10:25.174 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:23.744001+0000 mgr.y (mgr.14144) 71 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:10:25.175 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:23.744001+0000 mgr.y (mgr.14144) 71 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:10:25.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:24 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:23.744001+0000 mgr.y (mgr.14144) 71 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:10:26.161 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:25.508727+0000 mon.a (mon.0) 260 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-17T07:10:26.162 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:25.509367+0000 mon.a (mon.0) 261 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:10:26.162 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:10:25.509925+0000 mgr.y (mgr.14144) 72 : cephadm [INF] Deploying daemon osd.2 on smithi044 2024-09-17T07:10:26.162 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:25.508727+0000 mon.a (mon.0) 260 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-17T07:10:26.162 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:25.509367+0000 mon.a (mon.0) 261 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:10:26.162 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:10:25.509925+0000 mgr.y (mgr.14144) 72 : cephadm [INF] Deploying daemon osd.2 on smithi044 2024-09-17T07:10:26.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:25 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:25.508727+0000 mon.a (mon.0) 260 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-17T07:10:26.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:25 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:25.509367+0000 mon.a (mon.0) 261 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:10:26.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:25 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:10:25.509925+0000 mgr.y (mgr.14144) 72 : cephadm [INF] Deploying daemon osd.2 on smithi044 2024-09-17T07:10:27.205 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:25.744372+0000 mgr.y (mgr.14144) 73 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:10:27.205 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:25.744372+0000 mgr.y (mgr.14144) 73 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:10:27.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:26 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:25.744372+0000 mgr.y (mgr.14144) 73 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:10:29.010 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:28 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:27.744615+0000 mgr.y (mgr.14144) 74 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:10:29.010 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:28 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:27.744615+0000 mgr.y (mgr.14144) 74 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:10:29.264 INFO:teuthology.orchestra.run.smithi044.stdout:Created osd(s) 2 on host 'smithi044' 2024-09-17T07:10:29.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:28 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:27.744615+0000 mgr.y (mgr.14144) 74 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:10:29.870 DEBUG:teuthology.orchestra.run.smithi044:osd.2> sudo journalctl -f -n 0 -u ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.2.service 2024-09-17T07:10:29.875 INFO:tasks.cephadm:Deploying osd.3 on smithi044 with /dev/vg_nvme/lv_1... 2024-09-17T07:10:29.876 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-09-17T07:10:30.228 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:29.251953+0000 mon.a (mon.0) 262 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:10:30.228 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:29.255633+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:30.228 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:29.262214+0000 mon.a (mon.0) 264 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:30.228 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:29.251953+0000 mon.a (mon.0) 262 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:10:30.228 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:29.255633+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:30.228 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:29.262214+0000 mon.a (mon.0) 264 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:30.265 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:29.251953+0000 mon.a (mon.0) 262 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:10:30.265 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:29.255633+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:30.265 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:29.262214+0000 mon.a (mon.0) 264 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:31.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:30 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:29.744882+0000 mgr.y (mgr.14144) 75 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:10:31.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:29.744882+0000 mgr.y (mgr.14144) 75 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:10:31.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:29.744882+0000 mgr.y (mgr.14144) 75 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:10:32.557 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-09-17T07:10:32.557 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-09-17T07:10:32.557 INFO:teuthology.orchestra.run.smithi044.stderr: stderr: 10+0 records in 2024-09-17T07:10:32.557 INFO:teuthology.orchestra.run.smithi044.stderr:10+0 records out 2024-09-17T07:10:32.557 INFO:teuthology.orchestra.run.smithi044.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0123414 s, 850 MB/s 2024-09-17T07:10:32.557 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping successful for: 2024-09-17T07:10:32.558 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:10:32 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.2[34077]: debug 2024-09-17T07:10:32.374+0000 7f8ab141f080 -1 osd.2 0 log_to_monitors {default=true} 2024-09-17T07:10:33.574 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:31.745156+0000 mgr.y (mgr.14144) 76 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:10:33.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:32.314258+0000 mon.a (mon.0) 265 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:33.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:32.379515+0000 mon.a (mon.0) 266 : audit [INF] from='osd.2 [v2:172.21.15.44:6818/3871662332,v1:172.21.15.44:6819/3871662332]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-17T07:10:33.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:31.745156+0000 mgr.y (mgr.14144) 76 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:10:33.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:32.314258+0000 mon.a (mon.0) 265 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:33.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:32.379515+0000 mon.a (mon.0) 266 : audit [INF] from='osd.2 [v2:172.21.15.44:6818/3871662332,v1:172.21.15.44:6819/3871662332]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-17T07:10:33.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:31.745156+0000 mgr.y (mgr.14144) 76 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:10:33.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:32.314258+0000 mon.a (mon.0) 265 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:33.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:32.379515+0000 mon.a (mon.0) 266 : audit [INF] from='osd.2 [v2:172.21.15.44:6818/3871662332,v1:172.21.15.44:6819/3871662332]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-17T07:10:34.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:33.317068+0000 mon.a (mon.0) 267 : audit [INF] from='osd.2 [v2:172.21.15.44:6818/3871662332,v1:172.21.15.44:6819/3871662332]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-17T07:10:34.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:33.317166+0000 mon.a (mon.0) 268 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-09-17T07:10:34.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:33.317544+0000 mon.a (mon.0) 269 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:10:34.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:33.317879+0000 mon.a (mon.0) 270 : audit [INF] from='osd.2 [v2:172.21.15.44:6818/3871662332,v1:172.21.15.44:6819/3871662332]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-09-17T07:10:34.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:33.317068+0000 mon.a (mon.0) 267 : audit [INF] from='osd.2 [v2:172.21.15.44:6818/3871662332,v1:172.21.15.44:6819/3871662332]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-17T07:10:34.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:33.317166+0000 mon.a (mon.0) 268 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-09-17T07:10:34.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:33.317544+0000 mon.a (mon.0) 269 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:10:34.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:33.317879+0000 mon.a (mon.0) 270 : audit [INF] from='osd.2 [v2:172.21.15.44:6818/3871662332,v1:172.21.15.44:6819/3871662332]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-09-17T07:10:34.576 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:10:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.2[34077]: debug 2024-09-17T07:10:34.322+0000 7f8aa7990700 -1 osd.2 0 waiting for initial osdmap 2024-09-17T07:10:34.576 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:10:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.2[34077]: debug 2024-09-17T07:10:34.327+0000 7f8aa2108700 -1 osd.2 17 set_numa_affinity unable to identify public interface 'ens1f0' numa node: (0) Success 2024-09-17T07:10:34.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:33.317068+0000 mon.a (mon.0) 267 : audit [INF] from='osd.2 [v2:172.21.15.44:6818/3871662332,v1:172.21.15.44:6819/3871662332]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-17T07:10:34.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:33.317166+0000 mon.a (mon.0) 268 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-09-17T07:10:34.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:33.317544+0000 mon.a (mon.0) 269 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:10:34.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:33.317879+0000 mon.a (mon.0) 270 : audit [INF] from='osd.2 [v2:172.21.15.44:6818/3871662332,v1:172.21.15.44:6819/3871662332]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-09-17T07:10:34.757 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi044:vg_nvme/lv_1 2024-09-17T07:10:35.982 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:33.745563+0000 mgr.y (mgr.14144) 77 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:10:35.982 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:34.318076+0000 mon.a (mon.0) 271 : audit [INF] from='osd.2 [v2:172.21.15.44:6818/3871662332,v1:172.21.15.44:6819/3871662332]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-09-17T07:10:35.982 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:34.318920+0000 mon.a (mon.0) 272 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-09-17T07:10:35.982 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:34.319142+0000 mon.a (mon.0) 273 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:10:35.982 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:34.323945+0000 mon.a (mon.0) 274 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:10:35.982 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:35.323304+0000 mon.a (mon.0) 275 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:10:35.982 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:33.745563+0000 mgr.y (mgr.14144) 77 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:10:35.982 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:34.318076+0000 mon.a (mon.0) 271 : audit [INF] from='osd.2 [v2:172.21.15.44:6818/3871662332,v1:172.21.15.44:6819/3871662332]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-09-17T07:10:35.983 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:34.318920+0000 mon.a (mon.0) 272 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-09-17T07:10:35.983 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:34.319142+0000 mon.a (mon.0) 273 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:10:35.983 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:34.323945+0000 mon.a (mon.0) 274 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:10:35.983 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:35.323304+0000 mon.a (mon.0) 275 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:10:36.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:35 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:33.745563+0000 mgr.y (mgr.14144) 77 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:10:36.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:35 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:34.318076+0000 mon.a (mon.0) 271 : audit [INF] from='osd.2 [v2:172.21.15.44:6818/3871662332,v1:172.21.15.44:6819/3871662332]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-09-17T07:10:36.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:35 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:34.318920+0000 mon.a (mon.0) 272 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-09-17T07:10:36.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:35 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:34.319142+0000 mon.a (mon.0) 273 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:10:36.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:35 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:34.323945+0000 mon.a (mon.0) 274 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:10:36.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:35 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:35.323304+0000 mon.a (mon.0) 275 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:10:37.070 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:33.368524+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T07:10:37.070 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:33.368703+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T07:10:37.071 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:35.327996+0000 mon.a (mon.0) 276 : cluster [INF] osd.2 [v2:172.21.15.44:6818/3871662332,v1:172.21.15.44:6819/3871662332] boot 2024-09-17T07:10:37.071 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:35.328030+0000 mon.a (mon.0) 277 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2024-09-17T07:10:37.071 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:35.328537+0000 mon.a (mon.0) 278 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:10:37.071 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:35.328898+0000 mon.a (mon.0) 279 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2024-09-17T07:10:37.071 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:35.745834+0000 mgr.y (mgr.14144) 78 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:10:37.071 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:10:35.926709+0000 mgr.y (mgr.14144) 79 : cephadm [INF] Detected new or changed devices on smithi044 2024-09-17T07:10:37.071 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:35.933223+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:37.071 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:35.938633+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:37.071 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:35.939756+0000 mon.a (mon.0) 282 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:10:37.071 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:35.945597+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:37.071 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:36.336623+0000 mon.a (mon.0) 284 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2024-09-17T07:10:37.071 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:36.336684+0000 mon.a (mon.0) 285 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-09-17T07:10:37.071 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:36.337279+0000 mon.a (mon.0) 286 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-09-17T07:10:37.072 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:33.368524+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T07:10:37.072 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:33.368703+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T07:10:37.072 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:35.327996+0000 mon.a (mon.0) 276 : cluster [INF] osd.2 [v2:172.21.15.44:6818/3871662332,v1:172.21.15.44:6819/3871662332] boot 2024-09-17T07:10:37.072 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:35.328030+0000 mon.a (mon.0) 277 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2024-09-17T07:10:37.072 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:35.328537+0000 mon.a (mon.0) 278 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:10:37.072 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:35.328898+0000 mon.a (mon.0) 279 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2024-09-17T07:10:37.072 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:35.745834+0000 mgr.y (mgr.14144) 78 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:10:37.072 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:10:35.926709+0000 mgr.y (mgr.14144) 79 : cephadm [INF] Detected new or changed devices on smithi044 2024-09-17T07:10:37.072 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:35.933223+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:37.072 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:35.938633+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:37.072 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:35.939756+0000 mon.a (mon.0) 282 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:10:37.073 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:35.945597+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:37.073 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:36.336623+0000 mon.a (mon.0) 284 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2024-09-17T07:10:37.073 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:36.336684+0000 mon.a (mon.0) 285 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-09-17T07:10:37.073 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:36.337279+0000 mon.a (mon.0) 286 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-09-17T07:10:37.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:33.368524+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T07:10:37.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:33.368703+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T07:10:37.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:35.327996+0000 mon.a (mon.0) 276 : cluster [INF] osd.2 [v2:172.21.15.44:6818/3871662332,v1:172.21.15.44:6819/3871662332] boot 2024-09-17T07:10:37.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:35.328030+0000 mon.a (mon.0) 277 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2024-09-17T07:10:37.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:35.328537+0000 mon.a (mon.0) 278 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:10:37.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:35.328898+0000 mon.a (mon.0) 279 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2024-09-17T07:10:37.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:35.745834+0000 mgr.y (mgr.14144) 78 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T07:10:37.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:10:35.926709+0000 mgr.y (mgr.14144) 79 : cephadm [INF] Detected new or changed devices on smithi044 2024-09-17T07:10:37.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:35.933223+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:37.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:35.938633+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:37.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:35.939756+0000 mon.a (mon.0) 282 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:10:37.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:35.945597+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:37.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:36.336623+0000 mon.a (mon.0) 284 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2024-09-17T07:10:37.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:36.336684+0000 mon.a (mon.0) 285 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-09-17T07:10:37.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:36.337279+0000 mon.a (mon.0) 286 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-09-17T07:10:38.598 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:37.340499+0000 mon.a (mon.0) 287 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-09-17T07:10:38.598 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:37.340614+0000 mon.a (mon.0) 288 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-09-17T07:10:38.598 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:37.386576+0000 mgr.y (mgr.14144) 80 : audit [DBG] from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:10:38.598 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:37.387233+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:10:38.598 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:10:37.388190+0000 mgr.y (mgr.14144) 81 : cephadm [INF] Applying service osd.None on host smithi044... 2024-09-17T07:10:38.598 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:37.388365+0000 mon.a (mon.0) 290 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:10:38.598 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:37.388899+0000 mon.a (mon.0) 291 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:10:38.599 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:37.340499+0000 mon.a (mon.0) 287 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-09-17T07:10:38.599 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:37.340614+0000 mon.a (mon.0) 288 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-09-17T07:10:38.599 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:37.386576+0000 mgr.y (mgr.14144) 80 : audit [DBG] from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:10:38.599 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:37.387233+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:10:38.599 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:10:37.388190+0000 mgr.y (mgr.14144) 81 : cephadm [INF] Applying service osd.None on host smithi044... 2024-09-17T07:10:38.599 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:37.388365+0000 mon.a (mon.0) 290 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:10:38.599 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:37.388899+0000 mon.a (mon.0) 291 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:10:38.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:38 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:37.340499+0000 mon.a (mon.0) 287 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-09-17T07:10:38.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:38 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:37.340614+0000 mon.a (mon.0) 288 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-09-17T07:10:38.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:38 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:37.386576+0000 mgr.y (mgr.14144) 80 : audit [DBG] from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:10:38.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:38 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:37.387233+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:10:38.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:38 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:10:37.388190+0000 mgr.y (mgr.14144) 81 : cephadm [INF] Applying service osd.None on host smithi044... 2024-09-17T07:10:38.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:38 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:37.388365+0000 mon.a (mon.0) 290 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:10:38.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:38 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:37.388899+0000 mon.a (mon.0) 291 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:10:39.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:37.746124+0000 mgr.y (mgr.14144) 82 : cluster [DBG] pgmap v65: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:10:39.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:38.347413+0000 mon.a (mon.0) 292 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-09-17T07:10:39.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:37.746124+0000 mgr.y (mgr.14144) 82 : cluster [DBG] pgmap v65: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:10:39.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:38.347413+0000 mon.a (mon.0) 292 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-09-17T07:10:39.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:37.746124+0000 mgr.y (mgr.14144) 82 : cluster [DBG] pgmap v65: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:10:39.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:38.347413+0000 mon.a (mon.0) 292 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-09-17T07:10:40.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:40 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:39.603971+0000 mon.a (mon.0) 293 : audit [INF] from='client.? 172.21.15.44:0/1992271479' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72f5c16b-cdf4-45fa-a49b-93fed3cf8d07"}]: dispatch 2024-09-17T07:10:40.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:40 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:39.608462+0000 mon.a (mon.0) 294 : audit [INF] from='client.? 172.21.15.44:0/1992271479' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "72f5c16b-cdf4-45fa-a49b-93fed3cf8d07"}]': finished 2024-09-17T07:10:40.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:40 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:39.608513+0000 mon.a (mon.0) 295 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2024-09-17T07:10:40.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:40 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:39.608750+0000 mon.a (mon.0) 296 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:10:40.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:40 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:40.038917+0000 mon.a (mon.0) 297 : audit [DBG] from='client.? 172.21.15.44:0/1543036077' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:10:40.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:39.603971+0000 mon.a (mon.0) 293 : audit [INF] from='client.? 172.21.15.44:0/1992271479' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72f5c16b-cdf4-45fa-a49b-93fed3cf8d07"}]: dispatch 2024-09-17T07:10:40.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:39.608462+0000 mon.a (mon.0) 294 : audit [INF] from='client.? 172.21.15.44:0/1992271479' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "72f5c16b-cdf4-45fa-a49b-93fed3cf8d07"}]': finished 2024-09-17T07:10:40.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:39.608513+0000 mon.a (mon.0) 295 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2024-09-17T07:10:40.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:39.608750+0000 mon.a (mon.0) 296 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:10:40.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:40.038917+0000 mon.a (mon.0) 297 : audit [DBG] from='client.? 172.21.15.44:0/1543036077' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:10:40.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:39.603971+0000 mon.a (mon.0) 293 : audit [INF] from='client.? 172.21.15.44:0/1992271479' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72f5c16b-cdf4-45fa-a49b-93fed3cf8d07"}]: dispatch 2024-09-17T07:10:40.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:39.608462+0000 mon.a (mon.0) 294 : audit [INF] from='client.? 172.21.15.44:0/1992271479' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "72f5c16b-cdf4-45fa-a49b-93fed3cf8d07"}]': finished 2024-09-17T07:10:40.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:39.608513+0000 mon.a (mon.0) 295 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2024-09-17T07:10:40.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:39.608750+0000 mon.a (mon.0) 296 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:10:40.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:40.038917+0000 mon.a (mon.0) 297 : audit [DBG] from='client.? 172.21.15.44:0/1543036077' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:10:41.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:41 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:39.746392+0000 mgr.y (mgr.14144) 83 : cluster [DBG] pgmap v68: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:10:41.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:41 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:39.746392+0000 mgr.y (mgr.14144) 83 : cluster [DBG] pgmap v68: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:10:41.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:41 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:39.746392+0000 mgr.y (mgr.14144) 83 : cluster [DBG] pgmap v68: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:10:43.446 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:43 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:41.746827+0000 mgr.y (mgr.14144) 84 : cluster [DBG] pgmap v69: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:10:43.447 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:43 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:41.746827+0000 mgr.y (mgr.14144) 84 : cluster [DBG] pgmap v69: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:10:43.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:43 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:41.746827+0000 mgr.y (mgr.14144) 84 : cluster [DBG] pgmap v69: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:10:45.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:43.747244+0000 mgr.y (mgr.14144) 85 : cluster [DBG] pgmap v70: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:10:45.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:43.747244+0000 mgr.y (mgr.14144) 85 : cluster [DBG] pgmap v70: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:10:45.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:43.747244+0000 mgr.y (mgr.14144) 85 : cluster [DBG] pgmap v70: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:10:46.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:45.690895+0000 mon.a (mon.0) 298 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-17T07:10:46.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:45.691633+0000 mon.a (mon.0) 299 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:10:46.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:45.690895+0000 mon.a (mon.0) 298 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-17T07:10:46.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:45.691633+0000 mon.a (mon.0) 299 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:10:46.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:45.690895+0000 mon.a (mon.0) 298 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-17T07:10:46.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:45.691633+0000 mon.a (mon.0) 299 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:10:47.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:10:45.692225+0000 mgr.y (mgr.14144) 86 : cephadm [INF] Deploying daemon osd.3 on smithi044 2024-09-17T07:10:47.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:45.747559+0000 mgr.y (mgr.14144) 87 : cluster [DBG] pgmap v71: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:10:47.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:10:45.692225+0000 mgr.y (mgr.14144) 86 : cephadm [INF] Deploying daemon osd.3 on smithi044 2024-09-17T07:10:47.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:45.747559+0000 mgr.y (mgr.14144) 87 : cluster [DBG] pgmap v71: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:10:47.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:10:45.692225+0000 mgr.y (mgr.14144) 86 : cephadm [INF] Deploying daemon osd.3 on smithi044 2024-09-17T07:10:47.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:45.747559+0000 mgr.y (mgr.14144) 87 : cluster [DBG] pgmap v71: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:10:48.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:47.864401+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:10:48.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:47.865446+0000 mon.a (mon.0) 301 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:10:48.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:47.864401+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:10:48.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:47.865446+0000 mon.a (mon.0) 301 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:10:48.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:47.864401+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:10:48.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:47.865446+0000 mon.a (mon.0) 301 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:10:49.433 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:47.747957+0000 mgr.y (mgr.14144) 88 : cluster [DBG] pgmap v72: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:10:49.434 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:47.747957+0000 mgr.y (mgr.14144) 88 : cluster [DBG] pgmap v72: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:10:49.434 INFO:teuthology.orchestra.run.smithi044.stdout:Created osd(s) 3 on host 'smithi044' 2024-09-17T07:10:49.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:47.747957+0000 mgr.y (mgr.14144) 88 : cluster [DBG] pgmap v72: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:10:50.103 DEBUG:teuthology.orchestra.run.smithi044:osd.3> sudo journalctl -f -n 0 -u ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.3.service 2024-09-17T07:10:50.106 INFO:tasks.cephadm:Deploying osd.4 on smithi179 with /dev/vg_nvme/lv_4... 2024-09-17T07:10:50.106 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-09-17T07:10:50.407 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:49.418735+0000 mon.a (mon.0) 302 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:10:50.407 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:49.424210+0000 mon.a (mon.0) 303 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:50.408 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:49.431703+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:50.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:49.418735+0000 mon.a (mon.0) 302 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:10:50.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:49.424210+0000 mon.a (mon.0) 303 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:50.618 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:49.431703+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:50.618 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:49.418735+0000 mon.a (mon.0) 302 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:10:50.618 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:49.424210+0000 mon.a (mon.0) 303 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:50.618 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:49.431703+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:51.561 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:49.748337+0000 mgr.y (mgr.14144) 89 : cluster [DBG] pgmap v73: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:10:51.561 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:51.101053+0000 mon.a (mon.0) 305 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:51.677 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:49.748337+0000 mgr.y (mgr.14144) 89 : cluster [DBG] pgmap v73: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:10:51.677 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:51.101053+0000 mon.a (mon.0) 305 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:51.678 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:49.748337+0000 mgr.y (mgr.14144) 89 : cluster [DBG] pgmap v73: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:10:51.678 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:51.101053+0000 mon.a (mon.0) 305 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:52.072 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-09-17T07:10:52.072 INFO:teuthology.orchestra.run.smithi179.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-09-17T07:10:52.072 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10+0 records in 2024-09-17T07:10:52.072 INFO:teuthology.orchestra.run.smithi179.stderr:10+0 records out 2024-09-17T07:10:52.072 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0136333 s, 769 MB/s 2024-09-17T07:10:52.072 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping successful for: 2024-09-17T07:10:52.575 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:10:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.3[36511]: debug 2024-09-17T07:10:52.548+0000 7fe7987b3080 -1 osd.3 0 log_to_monitors {default=true} 2024-09-17T07:10:52.701 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi179:vg_nvme/lv_4 2024-09-17T07:10:53.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:51.748636+0000 mgr.y (mgr.14144) 90 : cluster [DBG] pgmap v74: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:10:53.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:52.552725+0000 mon.a (mon.0) 306 : audit [INF] from='osd.3 [v2:172.21.15.44:6826/905450488,v1:172.21.15.44:6827/905450488]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-17T07:10:53.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:53 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:51.748636+0000 mgr.y (mgr.14144) 90 : cluster [DBG] pgmap v74: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:10:54.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:53 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:52.552725+0000 mon.a (mon.0) 306 : audit [INF] from='osd.3 [v2:172.21.15.44:6826/905450488,v1:172.21.15.44:6827/905450488]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-17T07:10:54.687 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:53 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:51.748636+0000 mgr.y (mgr.14144) 90 : cluster [DBG] pgmap v74: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:10:54.687 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:53 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:52.552725+0000 mon.a (mon.0) 306 : audit [INF] from='osd.3 [v2:172.21.15.44:6826/905450488,v1:172.21.15.44:6827/905450488]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-17T07:10:54.687 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:10:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.3[36511]: debug 2024-09-17T07:10:54.385+0000 7fe78ed24700 -1 osd.3 0 waiting for initial osdmap 2024-09-17T07:10:54.687 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:10:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.3[36511]: debug 2024-09-17T07:10:54.393+0000 7fe787498700 -1 osd.3 24 set_numa_affinity unable to identify public interface 'ens1f0' numa node: (0) Success 2024-09-17T07:10:54.688 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:53.383667+0000 mon.a (mon.0) 307 : audit [INF] from='osd.3 [v2:172.21.15.44:6826/905450488,v1:172.21.15.44:6827/905450488]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-17T07:10:54.688 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:53.383736+0000 mon.a (mon.0) 308 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-09-17T07:10:54.688 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:53.384162+0000 mon.a (mon.0) 309 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:10:54.688 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:53.384509+0000 mon.a (mon.0) 310 : audit [INF] from='osd.3 [v2:172.21.15.44:6826/905450488,v1:172.21.15.44:6827/905450488]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-09-17T07:10:54.688 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:54.367037+0000 mon.a (mon.0) 311 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:54.688 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:54.369981+0000 mon.a (mon.0) 312 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:54.688 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:54.371189+0000 mon.a (mon.0) 313 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:10:54.688 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:54.374776+0000 mon.a (mon.0) 314 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:54.688 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:54.383043+0000 mon.a (mon.0) 315 : audit [INF] from='osd.3 [v2:172.21.15.44:6826/905450488,v1:172.21.15.44:6827/905450488]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-09-17T07:10:54.689 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:54.383138+0000 mon.a (mon.0) 316 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-09-17T07:10:54.689 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:54.383255+0000 mon.a (mon.0) 317 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:10:54.689 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:53.383667+0000 mon.a (mon.0) 307 : audit [INF] from='osd.3 [v2:172.21.15.44:6826/905450488,v1:172.21.15.44:6827/905450488]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-17T07:10:54.689 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:53.383736+0000 mon.a (mon.0) 308 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-09-17T07:10:54.689 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:53.384162+0000 mon.a (mon.0) 309 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:10:54.689 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:53.384509+0000 mon.a (mon.0) 310 : audit [INF] from='osd.3 [v2:172.21.15.44:6826/905450488,v1:172.21.15.44:6827/905450488]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-09-17T07:10:54.689 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:54.367037+0000 mon.a (mon.0) 311 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:54.689 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:54.369981+0000 mon.a (mon.0) 312 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:54.689 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:54.371189+0000 mon.a (mon.0) 313 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:10:54.689 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:54.374776+0000 mon.a (mon.0) 314 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:54.690 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:54.383043+0000 mon.a (mon.0) 315 : audit [INF] from='osd.3 [v2:172.21.15.44:6826/905450488,v1:172.21.15.44:6827/905450488]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-09-17T07:10:54.690 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:54.383138+0000 mon.a (mon.0) 316 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-09-17T07:10:54.690 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:54.383255+0000 mon.a (mon.0) 317 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:10:54.690 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:53.383667+0000 mon.a (mon.0) 307 : audit [INF] from='osd.3 [v2:172.21.15.44:6826/905450488,v1:172.21.15.44:6827/905450488]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-17T07:10:54.690 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:53.383736+0000 mon.a (mon.0) 308 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-09-17T07:10:54.690 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:53.384162+0000 mon.a (mon.0) 309 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:10:54.690 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:53.384509+0000 mon.a (mon.0) 310 : audit [INF] from='osd.3 [v2:172.21.15.44:6826/905450488,v1:172.21.15.44:6827/905450488]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-09-17T07:10:54.690 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:54.367037+0000 mon.a (mon.0) 311 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:54.690 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:54.369981+0000 mon.a (mon.0) 312 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:54.690 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:54.371189+0000 mon.a (mon.0) 313 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:10:54.691 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:54.374776+0000 mon.a (mon.0) 314 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:10:54.691 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:54.383043+0000 mon.a (mon.0) 315 : audit [INF] from='osd.3 [v2:172.21.15.44:6826/905450488,v1:172.21.15.44:6827/905450488]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-09-17T07:10:54.691 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:54.383138+0000 mon.a (mon.0) 316 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-09-17T07:10:54.691 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:54.383255+0000 mon.a (mon.0) 317 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:10:55.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:53.748965+0000 mgr.y (mgr.14144) 91 : cluster [DBG] pgmap v76: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:10:55.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:10:54.362807+0000 mgr.y (mgr.14144) 92 : cephadm [INF] Detected new or changed devices on smithi044 2024-09-17T07:10:55.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:54.386325+0000 mon.a (mon.0) 318 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:10:55.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:54.529316+0000 mgr.y (mgr.14144) 93 : audit [DBG] from='client.34187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:10:55.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:54.530019+0000 mon.a (mon.0) 319 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:10:55.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:10:54.531158+0000 mgr.y (mgr.14144) 94 : cephadm [INF] Applying service osd.None on host smithi179... 2024-09-17T07:10:55.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:54.531360+0000 mon.a (mon.0) 320 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:10:55.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:54.531860+0000 mon.a (mon.0) 321 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:10:55.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:55.386201+0000 mon.a (mon.0) 322 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:10:55.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:53.748965+0000 mgr.y (mgr.14144) 91 : cluster [DBG] pgmap v76: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:10:55.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:10:54.362807+0000 mgr.y (mgr.14144) 92 : cephadm [INF] Detected new or changed devices on smithi044 2024-09-17T07:10:55.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:54.386325+0000 mon.a (mon.0) 318 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:10:55.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:54.529316+0000 mgr.y (mgr.14144) 93 : audit [DBG] from='client.34187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:10:55.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:54.530019+0000 mon.a (mon.0) 319 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:10:55.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:10:54.531158+0000 mgr.y (mgr.14144) 94 : cephadm [INF] Applying service osd.None on host smithi179... 2024-09-17T07:10:55.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:54.531360+0000 mon.a (mon.0) 320 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:10:55.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:54.531860+0000 mon.a (mon.0) 321 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:10:55.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:55.386201+0000 mon.a (mon.0) 322 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:10:55.743 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:53.748965+0000 mgr.y (mgr.14144) 91 : cluster [DBG] pgmap v76: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:10:55.743 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:10:54.362807+0000 mgr.y (mgr.14144) 92 : cephadm [INF] Detected new or changed devices on smithi044 2024-09-17T07:10:55.743 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:54.386325+0000 mon.a (mon.0) 318 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:10:55.743 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:54.529316+0000 mgr.y (mgr.14144) 93 : audit [DBG] from='client.34187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:10:55.743 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:54.530019+0000 mon.a (mon.0) 319 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:10:55.743 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:10:54.531158+0000 mgr.y (mgr.14144) 94 : cephadm [INF] Applying service osd.None on host smithi179... 2024-09-17T07:10:55.743 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:54.531360+0000 mon.a (mon.0) 320 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:10:55.743 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:54.531860+0000 mon.a (mon.0) 321 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:10:55.744 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:55.386201+0000 mon.a (mon.0) 322 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:10:56.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:53.566981+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T07:10:56.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:53.567067+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T07:10:56.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:55.401322+0000 mon.a (mon.0) 323 : cluster [INF] osd.3 [v2:172.21.15.44:6826/905450488,v1:172.21.15.44:6827/905450488] boot 2024-09-17T07:10:56.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:55.401367+0000 mon.a (mon.0) 324 : cluster [DBG] osdmap e25: 4 total, 4 up, 4 in 2024-09-17T07:10:56.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:55.401885+0000 mon.a (mon.0) 325 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:10:56.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:53.566981+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T07:10:56.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:53.567067+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T07:10:56.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:55.401322+0000 mon.a (mon.0) 323 : cluster [INF] osd.3 [v2:172.21.15.44:6826/905450488,v1:172.21.15.44:6827/905450488] boot 2024-09-17T07:10:56.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:55.401367+0000 mon.a (mon.0) 324 : cluster [DBG] osdmap e25: 4 total, 4 up, 4 in 2024-09-17T07:10:56.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:55.401885+0000 mon.a (mon.0) 325 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:10:56.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:56 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:53.566981+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T07:10:56.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:56 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:53.567067+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T07:10:56.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:56 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:55.401322+0000 mon.a (mon.0) 323 : cluster [INF] osd.3 [v2:172.21.15.44:6826/905450488,v1:172.21.15.44:6827/905450488] boot 2024-09-17T07:10:56.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:56 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:55.401367+0000 mon.a (mon.0) 324 : cluster [DBG] osdmap e25: 4 total, 4 up, 4 in 2024-09-17T07:10:56.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:56 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:55.401885+0000 mon.a (mon.0) 325 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:10:57.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:55.749386+0000 mgr.y (mgr.14144) 95 : cluster [DBG] pgmap v79: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:10:57.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:56.412149+0000 mon.a (mon.0) 326 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2024-09-17T07:10:57.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:56.736973+0000 mon.b (mon.2) 6 : audit [INF] from='client.? 172.21.15.179:0/2648622901' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff638478-3d19-4b13-9123-4be3ad21a531"}]: dispatch 2024-09-17T07:10:57.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:56.737588+0000 mon.a (mon.0) 327 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff638478-3d19-4b13-9123-4be3ad21a531"}]: dispatch 2024-09-17T07:10:57.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:56.744580+0000 mon.a (mon.0) 328 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff638478-3d19-4b13-9123-4be3ad21a531"}]': finished 2024-09-17T07:10:57.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:56.744684+0000 mon.a (mon.0) 329 : cluster [DBG] osdmap e27: 5 total, 4 up, 5 in 2024-09-17T07:10:57.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:56.744857+0000 mon.a (mon.0) 330 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:10:57.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:10:57.145670+0000 mon.b (mon.2) 7 : audit [DBG] from='client.? 172.21.15.179:0/1684151219' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:10:57.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:55.749386+0000 mgr.y (mgr.14144) 95 : cluster [DBG] pgmap v79: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:10:57.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:56.412149+0000 mon.a (mon.0) 326 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2024-09-17T07:10:57.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:56.736973+0000 mon.b (mon.2) 6 : audit [INF] from='client.? 172.21.15.179:0/2648622901' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff638478-3d19-4b13-9123-4be3ad21a531"}]: dispatch 2024-09-17T07:10:57.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:56.737588+0000 mon.a (mon.0) 327 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff638478-3d19-4b13-9123-4be3ad21a531"}]: dispatch 2024-09-17T07:10:57.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:56.744580+0000 mon.a (mon.0) 328 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff638478-3d19-4b13-9123-4be3ad21a531"}]': finished 2024-09-17T07:10:57.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:56.744684+0000 mon.a (mon.0) 329 : cluster [DBG] osdmap e27: 5 total, 4 up, 5 in 2024-09-17T07:10:57.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:56.744857+0000 mon.a (mon.0) 330 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:10:57.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:10:57.145670+0000 mon.b (mon.2) 7 : audit [DBG] from='client.? 172.21.15.179:0/1684151219' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:10:57.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:55.749386+0000 mgr.y (mgr.14144) 95 : cluster [DBG] pgmap v79: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T07:10:57.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:56.412149+0000 mon.a (mon.0) 326 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2024-09-17T07:10:57.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:56.736973+0000 mon.b (mon.2) 6 : audit [INF] from='client.? 172.21.15.179:0/2648622901' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff638478-3d19-4b13-9123-4be3ad21a531"}]: dispatch 2024-09-17T07:10:57.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:56.737588+0000 mon.a (mon.0) 327 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff638478-3d19-4b13-9123-4be3ad21a531"}]: dispatch 2024-09-17T07:10:57.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:56.744580+0000 mon.a (mon.0) 328 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff638478-3d19-4b13-9123-4be3ad21a531"}]': finished 2024-09-17T07:10:57.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:56.744684+0000 mon.a (mon.0) 329 : cluster [DBG] osdmap e27: 5 total, 4 up, 5 in 2024-09-17T07:10:57.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:56.744857+0000 mon.a (mon.0) 330 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:10:57.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:10:57.145670+0000 mon.b (mon.2) 7 : audit [DBG] from='client.? 172.21.15.179:0/1684151219' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:10:59.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:10:59 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:57.749769+0000 mgr.y (mgr.14144) 96 : cluster [DBG] pgmap v82: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:10:59.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:10:59 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:57.749769+0000 mgr.y (mgr.14144) 96 : cluster [DBG] pgmap v82: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:10:59.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:10:59 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:57.749769+0000 mgr.y (mgr.14144) 96 : cluster [DBG] pgmap v82: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:11:00.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:00 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:10:59.750272+0000 mgr.y (mgr.14144) 97 : cluster [DBG] pgmap v83: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:11:01.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:10:59.750272+0000 mgr.y (mgr.14144) 97 : cluster [DBG] pgmap v83: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:11:01.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:10:59.750272+0000 mgr.y (mgr.14144) 97 : cluster [DBG] pgmap v83: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:11:03.031 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:01.750601+0000 mgr.y (mgr.14144) 98 : cluster [DBG] pgmap v84: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:11:03.031 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:02.387795+0000 mon.a (mon.0) 331 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-17T07:11:03.031 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:02.388597+0000 mon.a (mon.0) 332 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:11:03.031 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:11:02.389266+0000 mgr.y (mgr.14144) 99 : cephadm [INF] Deploying daemon osd.4 on smithi179 2024-09-17T07:11:03.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:01.750601+0000 mgr.y (mgr.14144) 98 : cluster [DBG] pgmap v84: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:11:03.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:02.387795+0000 mon.a (mon.0) 331 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-17T07:11:03.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:02.388597+0000 mon.a (mon.0) 332 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:11:03.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:11:02.389266+0000 mgr.y (mgr.14144) 99 : cephadm [INF] Deploying daemon osd.4 on smithi179 2024-09-17T07:11:03.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:01.750601+0000 mgr.y (mgr.14144) 98 : cluster [DBG] pgmap v84: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:11:03.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:02.387795+0000 mon.a (mon.0) 331 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-17T07:11:03.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:02.388597+0000 mon.a (mon.0) 332 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:11:03.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:11:02.389266+0000 mgr.y (mgr.14144) 99 : cephadm [INF] Deploying daemon osd.4 on smithi179 2024-09-17T07:11:05.031 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:04 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:03.750977+0000 mgr.y (mgr.14144) 100 : cluster [DBG] pgmap v85: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:11:05.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:04 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:03.750977+0000 mgr.y (mgr.14144) 100 : cluster [DBG] pgmap v85: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:11:05.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:04 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:03.750977+0000 mgr.y (mgr.14144) 100 : cluster [DBG] pgmap v85: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:11:05.866 INFO:teuthology.orchestra.run.smithi179.stdout:Created osd(s) 4 on host 'smithi179' 2024-09-17T07:11:06.503 DEBUG:teuthology.orchestra.run.smithi179:osd.4> sudo journalctl -f -n 0 -u ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.4.service 2024-09-17T07:11:06.506 INFO:tasks.cephadm:Deploying osd.5 on smithi179 with /dev/vg_nvme/lv_3... 2024-09-17T07:11:06.506 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-09-17T07:11:07.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:05.751408+0000 mgr.y (mgr.14144) 101 : cluster [DBG] pgmap v86: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:11:07.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:05.850792+0000 mon.a (mon.0) 333 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:11:07.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:05.855128+0000 mon.a (mon.0) 334 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:07.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:05.864269+0000 mon.a (mon.0) 335 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:07.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:05.751408+0000 mgr.y (mgr.14144) 101 : cluster [DBG] pgmap v86: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:11:07.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:05.850792+0000 mon.a (mon.0) 333 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:11:07.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:05.855128+0000 mon.a (mon.0) 334 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:07.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:05.864269+0000 mon.a (mon.0) 335 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:07.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:05.751408+0000 mgr.y (mgr.14144) 101 : cluster [DBG] pgmap v86: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:11:07.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:05.850792+0000 mon.a (mon.0) 333 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:11:07.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:05.855128+0000 mon.a (mon.0) 334 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:07.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:05.864269+0000 mon.a (mon.0) 335 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:09.061 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:07.751786+0000 mgr.y (mgr.14144) 102 : cluster [DBG] pgmap v87: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:11:09.062 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:11:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.4[26649]: debug 2024-09-17T07:11:08.960+0000 7fa96ae5d080 -1 osd.4 0 log_to_monitors {default=true} 2024-09-17T07:11:09.062 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-09-17T07:11:09.062 INFO:teuthology.orchestra.run.smithi179.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-09-17T07:11:09.062 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10+0 records in 2024-09-17T07:11:09.062 INFO:teuthology.orchestra.run.smithi179.stderr:10+0 records out 2024-09-17T07:11:09.062 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0147436 s, 711 MB/s 2024-09-17T07:11:09.062 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping successful for: 2024-09-17T07:11:09.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:07.751786+0000 mgr.y (mgr.14144) 102 : cluster [DBG] pgmap v87: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:11:09.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:07.751786+0000 mgr.y (mgr.14144) 102 : cluster [DBG] pgmap v87: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:11:09.772 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi179:vg_nvme/lv_3 2024-09-17T07:11:10.026 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:08.819927+0000 mon.a (mon.0) 336 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:10.027 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:08.964270+0000 mon.b (mon.2) 8 : audit [INF] from='osd.4 [v2:172.21.15.179:6800/2112811328,v1:172.21.15.179:6801/2112811328]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T07:11:10.027 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:08.964800+0000 mon.a (mon.0) 337 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T07:11:10.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:08.819927+0000 mon.a (mon.0) 336 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:10.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:08.964270+0000 mon.b (mon.2) 8 : audit [INF] from='osd.4 [v2:172.21.15.179:6800/2112811328,v1:172.21.15.179:6801/2112811328]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T07:11:10.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:08.964800+0000 mon.a (mon.0) 337 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T07:11:10.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:08.819927+0000 mon.a (mon.0) 336 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:10.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:08.964270+0000 mon.b (mon.2) 8 : audit [INF] from='osd.4 [v2:172.21.15.179:6800/2112811328,v1:172.21.15.179:6801/2112811328]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T07:11:10.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:08.964800+0000 mon.a (mon.0) 337 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T07:11:10.875 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:11:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.4[26649]: debug 2024-09-17T07:11:10.828+0000 7fa9613ce700 -1 osd.4 0 waiting for initial osdmap 2024-09-17T07:11:10.875 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:11:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.4[26649]: debug 2024-09-17T07:11:10.835+0000 7fa959b42700 -1 osd.4 29 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2024-09-17T07:11:10.875 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:09.752228+0000 mgr.y (mgr.14144) 103 : cluster [DBG] pgmap v88: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:11:10.875 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:09.821969+0000 mon.a (mon.0) 338 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-17T07:11:10.876 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:09.822062+0000 mon.a (mon.0) 339 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2024-09-17T07:11:10.876 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:09.822520+0000 mon.a (mon.0) 340 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:11:10.876 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:09.822521+0000 mon.b (mon.2) 9 : audit [INF] from='osd.4 [v2:172.21.15.179:6800/2112811328,v1:172.21.15.179:6801/2112811328]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:11:10.876 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:09.822938+0000 mon.a (mon.0) 341 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:11:11.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:09.752228+0000 mgr.y (mgr.14144) 103 : cluster [DBG] pgmap v88: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:11:11.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:09.821969+0000 mon.a (mon.0) 338 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-17T07:11:11.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:09.822062+0000 mon.a (mon.0) 339 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2024-09-17T07:11:11.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:09.822520+0000 mon.a (mon.0) 340 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:11:11.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:09.822521+0000 mon.b (mon.2) 9 : audit [INF] from='osd.4 [v2:172.21.15.179:6800/2112811328,v1:172.21.15.179:6801/2112811328]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:11:11.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:09.822938+0000 mon.a (mon.0) 341 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:11:11.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:09.752228+0000 mgr.y (mgr.14144) 103 : cluster [DBG] pgmap v88: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:11:11.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:09.821969+0000 mon.a (mon.0) 338 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-17T07:11:11.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:09.822062+0000 mon.a (mon.0) 339 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2024-09-17T07:11:11.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:09.822520+0000 mon.a (mon.0) 340 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:11:11.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:09.822521+0000 mon.b (mon.2) 9 : audit [INF] from='osd.4 [v2:172.21.15.179:6800/2112811328,v1:172.21.15.179:6801/2112811328]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:11:11.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:09.822938+0000 mon.a (mon.0) 341 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:11:11.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:11 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:10.823894+0000 mon.a (mon.0) 342 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-09-17T07:11:11.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:11 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:10.824019+0000 mon.a (mon.0) 343 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2024-09-17T07:11:11.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:11 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:10.824186+0000 mon.a (mon.0) 344 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:11:11.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:11 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:10.829678+0000 mon.a (mon.0) 345 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:11:12.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:10.823894+0000 mon.a (mon.0) 342 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-09-17T07:11:12.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:10.824019+0000 mon.a (mon.0) 343 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2024-09-17T07:11:12.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:10.824186+0000 mon.a (mon.0) 344 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:11:12.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:10.829678+0000 mon.a (mon.0) 345 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:11:12.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:10.823894+0000 mon.a (mon.0) 342 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-09-17T07:11:12.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:10.824019+0000 mon.a (mon.0) 343 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2024-09-17T07:11:12.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:10.824186+0000 mon.a (mon.0) 344 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:11:12.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:10.829678+0000 mon.a (mon.0) 345 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:11:13.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:12 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:09.923890+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T07:11:13.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:12 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:09.924015+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T07:11:13.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:12 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:11.752691+0000 mgr.y (mgr.14144) 104 : cluster [DBG] pgmap v91: 1 pgs: 1 peering; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:11:13.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:12 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:11.826033+0000 mon.a (mon.0) 346 : cluster [INF] osd.4 [v2:172.21.15.179:6800/2112811328,v1:172.21.15.179:6801/2112811328] boot 2024-09-17T07:11:13.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:12 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:11.826088+0000 mon.a (mon.0) 347 : cluster [DBG] osdmap e30: 5 total, 5 up, 5 in 2024-09-17T07:11:13.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:12 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:11.826279+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:11:13.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:12 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:12.676332+0000 mon.a (mon.0) 349 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:13.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:12 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:12.826340+0000 mon.a (mon.0) 350 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2024-09-17T07:11:13.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:09.923890+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T07:11:13.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:09.924015+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T07:11:13.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:11.752691+0000 mgr.y (mgr.14144) 104 : cluster [DBG] pgmap v91: 1 pgs: 1 peering; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:11:13.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:11.826033+0000 mon.a (mon.0) 346 : cluster [INF] osd.4 [v2:172.21.15.179:6800/2112811328,v1:172.21.15.179:6801/2112811328] boot 2024-09-17T07:11:13.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:11.826088+0000 mon.a (mon.0) 347 : cluster [DBG] osdmap e30: 5 total, 5 up, 5 in 2024-09-17T07:11:13.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:11.826279+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:11:13.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:12.676332+0000 mon.a (mon.0) 349 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:13.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:12.826340+0000 mon.a (mon.0) 350 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2024-09-17T07:11:13.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:09.923890+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T07:11:13.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:09.924015+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T07:11:13.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:11.752691+0000 mgr.y (mgr.14144) 104 : cluster [DBG] pgmap v91: 1 pgs: 1 peering; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T07:11:13.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:11.826033+0000 mon.a (mon.0) 346 : cluster [INF] osd.4 [v2:172.21.15.179:6800/2112811328,v1:172.21.15.179:6801/2112811328] boot 2024-09-17T07:11:13.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:11.826088+0000 mon.a (mon.0) 347 : cluster [DBG] osdmap e30: 5 total, 5 up, 5 in 2024-09-17T07:11:13.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:11.826279+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:11:13.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:12.676332+0000 mon.a (mon.0) 349 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:13.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:12.826340+0000 mon.a (mon.0) 350 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2024-09-17T07:11:14.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:13 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:11:12.671734+0000 mgr.y (mgr.14144) 105 : cephadm [INF] Detected new or changed devices on smithi179 2024-09-17T07:11:14.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:13 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:12.891242+0000 mon.a (mon.0) 351 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:14.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:13 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:12.892916+0000 mon.a (mon.0) 352 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:11:14.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:13 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:12.898072+0000 mon.a (mon.0) 353 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:14.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:13 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:13.582245+0000 mon.a (mon.0) 354 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:11:14.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:13 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:13.583895+0000 mon.a (mon.0) 355 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:11:14.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:13 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:13.584713+0000 mon.a (mon.0) 356 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:11:14.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:13 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:13.827119+0000 mon.a (mon.0) 357 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2024-09-17T07:11:14.226 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:11:12.671734+0000 mgr.y (mgr.14144) 105 : cephadm [INF] Detected new or changed devices on smithi179 2024-09-17T07:11:14.227 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:12.891242+0000 mon.a (mon.0) 351 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:14.227 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:12.892916+0000 mon.a (mon.0) 352 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:11:14.227 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:12.898072+0000 mon.a (mon.0) 353 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:14.227 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:13.582245+0000 mon.a (mon.0) 354 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:11:14.227 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:13.583895+0000 mon.a (mon.0) 355 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:11:14.227 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:13.584713+0000 mon.a (mon.0) 356 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:11:14.227 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:13.827119+0000 mon.a (mon.0) 357 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2024-09-17T07:11:14.227 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:11:12.671734+0000 mgr.y (mgr.14144) 105 : cephadm [INF] Detected new or changed devices on smithi179 2024-09-17T07:11:14.227 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:12.891242+0000 mon.a (mon.0) 351 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:14.227 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:12.892916+0000 mon.a (mon.0) 352 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:11:14.227 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:12.898072+0000 mon.a (mon.0) 353 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:14.228 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:13.582245+0000 mon.a (mon.0) 354 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:11:14.228 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:13.583895+0000 mon.a (mon.0) 355 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:11:14.228 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:13.584713+0000 mon.a (mon.0) 356 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:11:14.228 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:13.827119+0000 mon.a (mon.0) 357 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2024-09-17T07:11:15.212 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:14 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:13.581355+0000 mgr.y (mgr.14144) 106 : audit [DBG] from='client.34214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:11:15.213 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:14 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:11:13.583591+0000 mgr.y (mgr.14144) 107 : cephadm [INF] Applying service osd.None on host smithi179... 2024-09-17T07:11:15.213 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:14 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:13.753020+0000 mgr.y (mgr.14144) 108 : cluster [DBG] pgmap v94: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:11:15.238 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:13.581355+0000 mgr.y (mgr.14144) 106 : audit [DBG] from='client.34214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:11:15.238 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:11:13.583591+0000 mgr.y (mgr.14144) 107 : cephadm [INF] Applying service osd.None on host smithi179... 2024-09-17T07:11:15.238 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:13.753020+0000 mgr.y (mgr.14144) 108 : cluster [DBG] pgmap v94: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:11:15.238 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:13.581355+0000 mgr.y (mgr.14144) 106 : audit [DBG] from='client.34214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:11:15.238 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:11:13.583591+0000 mgr.y (mgr.14144) 107 : cephadm [INF] Applying service osd.None on host smithi179... 2024-09-17T07:11:15.238 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:13.753020+0000 mgr.y (mgr.14144) 108 : cluster [DBG] pgmap v94: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:11:16.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:15.730124+0000 mon.b (mon.2) 10 : audit [INF] from='client.? 172.21.15.179:0/475022323' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83cd743c-9dcb-4027-8857-34773603b77c"}]: dispatch 2024-09-17T07:11:16.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:15.730595+0000 mon.a (mon.0) 358 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83cd743c-9dcb-4027-8857-34773603b77c"}]: dispatch 2024-09-17T07:11:16.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:15.734454+0000 mon.a (mon.0) 359 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "83cd743c-9dcb-4027-8857-34773603b77c"}]': finished 2024-09-17T07:11:16.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:15.734501+0000 mon.a (mon.0) 360 : cluster [DBG] osdmap e33: 6 total, 5 up, 6 in 2024-09-17T07:11:16.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:15.734624+0000 mon.a (mon.0) 361 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:11:16.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:15.730124+0000 mon.b (mon.2) 10 : audit [INF] from='client.? 172.21.15.179:0/475022323' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83cd743c-9dcb-4027-8857-34773603b77c"}]: dispatch 2024-09-17T07:11:16.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:15.730595+0000 mon.a (mon.0) 358 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83cd743c-9dcb-4027-8857-34773603b77c"}]: dispatch 2024-09-17T07:11:16.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:15.734454+0000 mon.a (mon.0) 359 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "83cd743c-9dcb-4027-8857-34773603b77c"}]': finished 2024-09-17T07:11:16.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:15.734501+0000 mon.a (mon.0) 360 : cluster [DBG] osdmap e33: 6 total, 5 up, 6 in 2024-09-17T07:11:16.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:15.734624+0000 mon.a (mon.0) 361 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:11:16.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:15.730124+0000 mon.b (mon.2) 10 : audit [INF] from='client.? 172.21.15.179:0/475022323' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83cd743c-9dcb-4027-8857-34773603b77c"}]: dispatch 2024-09-17T07:11:16.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:15.730595+0000 mon.a (mon.0) 358 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83cd743c-9dcb-4027-8857-34773603b77c"}]: dispatch 2024-09-17T07:11:16.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:15.734454+0000 mon.a (mon.0) 359 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "83cd743c-9dcb-4027-8857-34773603b77c"}]': finished 2024-09-17T07:11:16.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:15.734501+0000 mon.a (mon.0) 360 : cluster [DBG] osdmap e33: 6 total, 5 up, 6 in 2024-09-17T07:11:16.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:15.734624+0000 mon.a (mon.0) 361 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:11:17.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:15.753303+0000 mgr.y (mgr.14144) 109 : cluster [DBG] pgmap v97: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:11:17.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:16.140315+0000 mon.b (mon.2) 11 : audit [DBG] from='client.? 172.21.15.179:0/870202350' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:11:17.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:15.753303+0000 mgr.y (mgr.14144) 109 : cluster [DBG] pgmap v97: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:11:17.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:16.140315+0000 mon.b (mon.2) 11 : audit [DBG] from='client.? 172.21.15.179:0/870202350' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:11:17.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:16 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:15.753303+0000 mgr.y (mgr.14144) 109 : cluster [DBG] pgmap v97: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:11:17.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:16 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:16.140315+0000 mon.b (mon.2) 11 : audit [DBG] from='client.? 172.21.15.179:0/870202350' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:11:18.991 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:18 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:17.753658+0000 mgr.y (mgr.14144) 110 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:11:19.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:17.753658+0000 mgr.y (mgr.14144) 110 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:11:19.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:17.753658+0000 mgr.y (mgr.14144) 110 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:11:20.939 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:19.754142+0000 mgr.y (mgr.14144) 111 : cluster [DBG] pgmap v99: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:11:21.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:19.754142+0000 mgr.y (mgr.14144) 111 : cluster [DBG] pgmap v99: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:11:21.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:19.754142+0000 mgr.y (mgr.14144) 111 : cluster [DBG] pgmap v99: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:11:21.871 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:21 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:21.527688+0000 mon.a (mon.0) 362 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-17T07:11:21.871 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:21 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:21.528555+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:11:21.871 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:21 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:11:21.529271+0000 mgr.y (mgr.14144) 112 : cephadm [INF] Deploying daemon osd.5 on smithi179 2024-09-17T07:11:22.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:21 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:21.527688+0000 mon.a (mon.0) 362 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-17T07:11:22.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:21 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:21.528555+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:11:22.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:21 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:11:21.529271+0000 mgr.y (mgr.14144) 112 : cephadm [INF] Deploying daemon osd.5 on smithi179 2024-09-17T07:11:22.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:21 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:21.527688+0000 mon.a (mon.0) 362 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-17T07:11:22.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:21 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:21.528555+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:11:22.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:21 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:11:21.529271+0000 mgr.y (mgr.14144) 112 : cephadm [INF] Deploying daemon osd.5 on smithi179 2024-09-17T07:11:22.966 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:21.754619+0000 mgr.y (mgr.14144) 113 : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:11:23.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:22 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:21.754619+0000 mgr.y (mgr.14144) 113 : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:11:23.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:22 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:21.754619+0000 mgr.y (mgr.14144) 113 : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:11:24.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:24 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:23.754938+0000 mgr.y (mgr.14144) 114 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:11:25.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:23.754938+0000 mgr.y (mgr.14144) 114 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:11:25.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:23.754938+0000 mgr.y (mgr.14144) 114 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:11:25.166 INFO:teuthology.orchestra.run.smithi179.stdout:Created osd(s) 5 on host 'smithi179' 2024-09-17T07:11:25.820 DEBUG:teuthology.orchestra.run.smithi179:osd.5> sudo journalctl -f -n 0 -u ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.5.service 2024-09-17T07:11:25.823 INFO:tasks.cephadm:Deploying osd.6 on smithi179 with /dev/vg_nvme/lv_2... 2024-09-17T07:11:25.823 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-09-17T07:11:26.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:25.155067+0000 mon.a (mon.0) 364 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:11:26.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:25.159006+0000 mon.a (mon.0) 365 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:26.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:25.163355+0000 mon.a (mon.0) 366 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:26.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:25.155067+0000 mon.a (mon.0) 364 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:11:26.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:25.159006+0000 mon.a (mon.0) 365 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:26.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:25.163355+0000 mon.a (mon.0) 366 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:26.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:25 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:25.155067+0000 mon.a (mon.0) 364 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:11:26.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:25 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:25.159006+0000 mon.a (mon.0) 365 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:26.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:25 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:25.163355+0000 mon.a (mon.0) 366 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:27.435 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:27 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:25.755393+0000 mgr.y (mgr.14144) 115 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:11:27.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:25.755393+0000 mgr.y (mgr.14144) 115 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:11:27.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:25.755393+0000 mgr.y (mgr.14144) 115 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:11:28.468 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-09-17T07:11:28.468 INFO:teuthology.orchestra.run.smithi179.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-09-17T07:11:28.468 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10+0 records in 2024-09-17T07:11:28.468 INFO:teuthology.orchestra.run.smithi179.stderr:10+0 records out 2024-09-17T07:11:28.468 INFO:teuthology.orchestra.run.smithi179.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.014464 s, 725 MB/s 2024-09-17T07:11:28.468 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 2024-09-17T07:11:28.469 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping successful for: 2024-09-17T07:11:28.469 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:11:28 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.5[29018]: debug 2024-09-17T07:11:28.245+0000 7f7e80f97080 -1 osd.5 0 log_to_monitors {default=true} 2024-09-17T07:11:29.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:27.755754+0000 mgr.y (mgr.14144) 116 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:11:29.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:28.249968+0000 mon.b (mon.2) 12 : audit [INF] from='osd.5 [v2:172.21.15.179:6808/3243120327,v1:172.21.15.179:6809/3243120327]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T07:11:29.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:28.250677+0000 mon.a (mon.0) 367 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T07:11:29.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:28.271049+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:29.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:27.755754+0000 mgr.y (mgr.14144) 116 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:11:29.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:28.249968+0000 mon.b (mon.2) 12 : audit [INF] from='osd.5 [v2:172.21.15.179:6808/3243120327,v1:172.21.15.179:6809/3243120327]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T07:11:29.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:28.250677+0000 mon.a (mon.0) 367 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T07:11:29.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:28.271049+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:29.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:27.755754+0000 mgr.y (mgr.14144) 116 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:11:29.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:28.249968+0000 mon.b (mon.2) 12 : audit [INF] from='osd.5 [v2:172.21.15.179:6808/3243120327,v1:172.21.15.179:6809/3243120327]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T07:11:29.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:28.250677+0000 mon.a (mon.0) 367 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T07:11:29.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:28.271049+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:29.731 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi179:vg_nvme/lv_2 2024-09-17T07:11:30.408 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:11:30 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.5[29018]: debug 2024-09-17T07:11:30.183+0000 7f7e77508700 -1 osd.5 0 waiting for initial osdmap 2024-09-17T07:11:30.408 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:11:30 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.5[29018]: debug 2024-09-17T07:11:30.188+0000 7f7e71c80700 -1 osd.5 35 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2024-09-17T07:11:30.408 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:30 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:29.175231+0000 mon.a (mon.0) 369 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T07:11:30.408 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:30 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:29.175348+0000 mon.a (mon.0) 370 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2024-09-17T07:11:30.408 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:30 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:29.175518+0000 mon.a (mon.0) 371 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:11:30.408 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:30 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:29.175802+0000 mon.b (mon.2) 13 : audit [INF] from='osd.5 [v2:172.21.15.179:6808/3243120327,v1:172.21.15.179:6809/3243120327]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:11:30.408 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:30 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:29.176179+0000 mon.a (mon.0) 372 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:11:30.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:29.175231+0000 mon.a (mon.0) 369 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T07:11:30.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:29.175348+0000 mon.a (mon.0) 370 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2024-09-17T07:11:30.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:29.175518+0000 mon.a (mon.0) 371 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:11:30.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:29.175802+0000 mon.b (mon.2) 13 : audit [INF] from='osd.5 [v2:172.21.15.179:6808/3243120327,v1:172.21.15.179:6809/3243120327]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:11:30.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:29.176179+0000 mon.a (mon.0) 372 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:11:30.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:29.175231+0000 mon.a (mon.0) 369 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T07:11:30.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:29.175348+0000 mon.a (mon.0) 370 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2024-09-17T07:11:30.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:29.175518+0000 mon.a (mon.0) 371 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:11:30.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:29.175802+0000 mon.b (mon.2) 13 : audit [INF] from='osd.5 [v2:172.21.15.179:6808/3243120327,v1:172.21.15.179:6809/3243120327]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:11:30.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:29.176179+0000 mon.a (mon.0) 372 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:11:31.574 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:29.756190+0000 mgr.y (mgr.14144) 117 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:11:31.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:30.179752+0000 mon.a (mon.0) 373 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-09-17T07:11:31.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:30.179884+0000 mon.a (mon.0) 374 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2024-09-17T07:11:31.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:30.180407+0000 mon.a (mon.0) 375 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:11:31.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:30.184807+0000 mon.a (mon.0) 376 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:11:31.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:31.178140+0000 mon.a (mon.0) 377 : cluster [INF] osd.5 [v2:172.21.15.179:6808/3243120327,v1:172.21.15.179:6809/3243120327] boot 2024-09-17T07:11:31.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:31.178191+0000 mon.a (mon.0) 378 : cluster [DBG] osdmap e36: 6 total, 6 up, 6 in 2024-09-17T07:11:31.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:31.178431+0000 mon.a (mon.0) 379 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:11:31.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:29.756190+0000 mgr.y (mgr.14144) 117 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:11:31.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:30.179752+0000 mon.a (mon.0) 373 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-09-17T07:11:31.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:30.179884+0000 mon.a (mon.0) 374 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2024-09-17T07:11:31.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:30.180407+0000 mon.a (mon.0) 375 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:11:31.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:30.184807+0000 mon.a (mon.0) 376 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:11:31.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:31.178140+0000 mon.a (mon.0) 377 : cluster [INF] osd.5 [v2:172.21.15.179:6808/3243120327,v1:172.21.15.179:6809/3243120327] boot 2024-09-17T07:11:31.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:31.178191+0000 mon.a (mon.0) 378 : cluster [DBG] osdmap e36: 6 total, 6 up, 6 in 2024-09-17T07:11:31.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:31.178431+0000 mon.a (mon.0) 379 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:11:31.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:29.756190+0000 mgr.y (mgr.14144) 117 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T07:11:31.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:30.179752+0000 mon.a (mon.0) 373 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-09-17T07:11:31.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:30.179884+0000 mon.a (mon.0) 374 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2024-09-17T07:11:31.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:30.180407+0000 mon.a (mon.0) 375 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:11:31.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:30.184807+0000 mon.a (mon.0) 376 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:11:31.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:31.178140+0000 mon.a (mon.0) 377 : cluster [INF] osd.5 [v2:172.21.15.179:6808/3243120327,v1:172.21.15.179:6809/3243120327] boot 2024-09-17T07:11:31.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:31.178191+0000 mon.a (mon.0) 378 : cluster [DBG] osdmap e36: 6 total, 6 up, 6 in 2024-09-17T07:11:31.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:31.178431+0000 mon.a (mon.0) 379 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:11:32.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:29.294023+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T07:11:32.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:29.294194+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T07:11:32.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:32.182388+0000 mon.a (mon.0) 380 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2024-09-17T07:11:32.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:32 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:29.294023+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T07:11:32.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:32 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:29.294194+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T07:11:32.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:32 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:32.182388+0000 mon.a (mon.0) 380 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2024-09-17T07:11:32.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:32 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:29.294023+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T07:11:32.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:32 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:29.294194+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T07:11:32.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:32 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:32.182388+0000 mon.a (mon.0) 380 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2024-09-17T07:11:33.489 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:31.756612+0000 mgr.y (mgr.14144) 118 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:11:33.489 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:32.757368+0000 mon.a (mon.0) 381 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:33.489 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:32.763801+0000 mon.a (mon.0) 382 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:33.489 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:32.765673+0000 mon.a (mon.0) 383 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:11:33.489 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:32.772983+0000 mon.a (mon.0) 384 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:33.490 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:32.998429+0000 mon.a (mon.0) 385 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:11:33.490 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:33.000174+0000 mon.a (mon.0) 386 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:11:33.490 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:33.000934+0000 mon.a (mon.0) 387 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:11:33.490 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:33.186279+0000 mon.a (mon.0) 388 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2024-09-17T07:11:33.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:31.756612+0000 mgr.y (mgr.14144) 118 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:11:33.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:32.757368+0000 mon.a (mon.0) 381 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:33.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:32.763801+0000 mon.a (mon.0) 382 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:33.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:32.765673+0000 mon.a (mon.0) 383 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:11:33.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:32.772983+0000 mon.a (mon.0) 384 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:33.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:32.998429+0000 mon.a (mon.0) 385 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:11:33.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:33.000174+0000 mon.a (mon.0) 386 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:11:33.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:33.000934+0000 mon.a (mon.0) 387 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:11:33.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:33.186279+0000 mon.a (mon.0) 388 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2024-09-17T07:11:33.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:31.756612+0000 mgr.y (mgr.14144) 118 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:11:33.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:32.757368+0000 mon.a (mon.0) 381 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:33.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:32.763801+0000 mon.a (mon.0) 382 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:33.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:32.765673+0000 mon.a (mon.0) 383 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:11:33.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:32.772983+0000 mon.a (mon.0) 384 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:33.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:32.998429+0000 mon.a (mon.0) 385 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:11:33.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:33.000174+0000 mon.a (mon.0) 386 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:11:33.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:33.000934+0000 mon.a (mon.0) 387 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:11:33.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:33.186279+0000 mon.a (mon.0) 388 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2024-09-17T07:11:34.337 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:11:32.751080+0000 mgr.y (mgr.14144) 119 : cephadm [INF] Detected new or changed devices on smithi179 2024-09-17T07:11:34.337 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:32.997550+0000 mgr.y (mgr.14144) 120 : audit [DBG] from='client.34241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:11:34.337 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:11:32.999897+0000 mgr.y (mgr.14144) 121 : cephadm [INF] Applying service osd.None on host smithi179... 2024-09-17T07:11:34.574 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:11:32.751080+0000 mgr.y (mgr.14144) 119 : cephadm [INF] Detected new or changed devices on smithi179 2024-09-17T07:11:34.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:32.997550+0000 mgr.y (mgr.14144) 120 : audit [DBG] from='client.34241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:11:34.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:11:32.999897+0000 mgr.y (mgr.14144) 121 : cephadm [INF] Applying service osd.None on host smithi179... 2024-09-17T07:11:34.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:11:32.751080+0000 mgr.y (mgr.14144) 119 : cephadm [INF] Detected new or changed devices on smithi179 2024-09-17T07:11:34.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:32.997550+0000 mgr.y (mgr.14144) 120 : audit [DBG] from='client.34241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:11:34.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:11:32.999897+0000 mgr.y (mgr.14144) 121 : cephadm [INF] Applying service osd.None on host smithi179... 2024-09-17T07:11:35.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:33.756972+0000 mgr.y (mgr.14144) 122 : cluster [DBG] pgmap v111: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:11:35.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:35.051235+0000 mon.b (mon.2) 14 : audit [INF] from='client.? 172.21.15.179:0/959056850' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4e608f05-0fef-407f-aa40-ded55acd0bb3"}]: dispatch 2024-09-17T07:11:35.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:35.051965+0000 mon.a (mon.0) 389 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4e608f05-0fef-407f-aa40-ded55acd0bb3"}]: dispatch 2024-09-17T07:11:35.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:35.057065+0000 mon.a (mon.0) 390 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4e608f05-0fef-407f-aa40-ded55acd0bb3"}]': finished 2024-09-17T07:11:35.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:35.057163+0000 mon.a (mon.0) 391 : cluster [DBG] osdmap e39: 7 total, 6 up, 7 in 2024-09-17T07:11:35.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:35.057317+0000 mon.a (mon.0) 392 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:11:35.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:33.756972+0000 mgr.y (mgr.14144) 122 : cluster [DBG] pgmap v111: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:11:35.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:35.051235+0000 mon.b (mon.2) 14 : audit [INF] from='client.? 172.21.15.179:0/959056850' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4e608f05-0fef-407f-aa40-ded55acd0bb3"}]: dispatch 2024-09-17T07:11:35.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:35.051965+0000 mon.a (mon.0) 389 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4e608f05-0fef-407f-aa40-ded55acd0bb3"}]: dispatch 2024-09-17T07:11:35.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:35.057065+0000 mon.a (mon.0) 390 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4e608f05-0fef-407f-aa40-ded55acd0bb3"}]': finished 2024-09-17T07:11:35.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:35.057163+0000 mon.a (mon.0) 391 : cluster [DBG] osdmap e39: 7 total, 6 up, 7 in 2024-09-17T07:11:35.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:35.057317+0000 mon.a (mon.0) 392 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:11:35.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:35 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:33.756972+0000 mgr.y (mgr.14144) 122 : cluster [DBG] pgmap v111: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:11:35.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:35 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:35.051235+0000 mon.b (mon.2) 14 : audit [INF] from='client.? 172.21.15.179:0/959056850' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4e608f05-0fef-407f-aa40-ded55acd0bb3"}]: dispatch 2024-09-17T07:11:35.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:35 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:35.051965+0000 mon.a (mon.0) 389 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4e608f05-0fef-407f-aa40-ded55acd0bb3"}]: dispatch 2024-09-17T07:11:35.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:35 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:35.057065+0000 mon.a (mon.0) 390 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4e608f05-0fef-407f-aa40-ded55acd0bb3"}]': finished 2024-09-17T07:11:35.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:35 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:35.057163+0000 mon.a (mon.0) 391 : cluster [DBG] osdmap e39: 7 total, 6 up, 7 in 2024-09-17T07:11:35.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:35 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:35.057317+0000 mon.a (mon.0) 392 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:11:36.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:35.476842+0000 mon.b (mon.2) 15 : audit [DBG] from='client.? 172.21.15.179:0/2329466249' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:11:36.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:35.476842+0000 mon.b (mon.2) 15 : audit [DBG] from='client.? 172.21.15.179:0/2329466249' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:11:36.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:35.476842+0000 mon.b (mon.2) 15 : audit [DBG] from='client.? 172.21.15.179:0/2329466249' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:11:37.574 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:35.757418+0000 mgr.y (mgr.14144) 123 : cluster [DBG] pgmap v113: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:11:37.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:35.757418+0000 mgr.y (mgr.14144) 123 : cluster [DBG] pgmap v113: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:11:37.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:37 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:35.757418+0000 mgr.y (mgr.14144) 123 : cluster [DBG] pgmap v113: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:11:39.547 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:37.757795+0000 mgr.y (mgr.14144) 124 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:11:39.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:37.757795+0000 mgr.y (mgr.14144) 124 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:11:39.574 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:37.757795+0000 mgr.y (mgr.14144) 124 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:11:40.937 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:40 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:39.758215+0000 mgr.y (mgr.14144) 125 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:11:41.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:39.758215+0000 mgr.y (mgr.14144) 125 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:11:41.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:39.758215+0000 mgr.y (mgr.14144) 125 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:11:41.821 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:41 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:40.961709+0000 mon.a (mon.0) 393 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-17T07:11:41.822 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:41 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:40.962580+0000 mon.a (mon.0) 394 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:11:41.822 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:41 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:11:40.963348+0000 mgr.y (mgr.14144) 126 : cephadm [INF] Deploying daemon osd.6 on smithi179 2024-09-17T07:11:42.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:41 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:40.961709+0000 mon.a (mon.0) 393 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-17T07:11:42.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:41 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:40.962580+0000 mon.a (mon.0) 394 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:11:42.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:41 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:11:40.963348+0000 mgr.y (mgr.14144) 126 : cephadm [INF] Deploying daemon osd.6 on smithi179 2024-09-17T07:11:42.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:41 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:40.961709+0000 mon.a (mon.0) 393 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-17T07:11:42.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:41 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:40.962580+0000 mon.a (mon.0) 394 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:11:42.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:41 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:11:40.963348+0000 mgr.y (mgr.14144) 126 : cephadm [INF] Deploying daemon osd.6 on smithi179 2024-09-17T07:11:42.913 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:41.758750+0000 mgr.y (mgr.14144) 127 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:11:43.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:41.758750+0000 mgr.y (mgr.14144) 127 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:11:43.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:41.758750+0000 mgr.y (mgr.14144) 127 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:11:44.533 INFO:teuthology.orchestra.run.smithi179.stdout:Created osd(s) 6 on host 'smithi179' 2024-09-17T07:11:45.067 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:43.759095+0000 mgr.y (mgr.14144) 128 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:11:45.067 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:44.522070+0000 mon.a (mon.0) 395 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:11:45.067 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:44.526390+0000 mon.a (mon.0) 396 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:45.067 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:44.531374+0000 mon.a (mon.0) 397 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:45.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:43.759095+0000 mgr.y (mgr.14144) 128 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:11:45.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:44.522070+0000 mon.a (mon.0) 395 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:11:45.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:44.526390+0000 mon.a (mon.0) 396 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:45.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:44.531374+0000 mon.a (mon.0) 397 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:45.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:43.759095+0000 mgr.y (mgr.14144) 128 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:11:45.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:44.522070+0000 mon.a (mon.0) 395 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:11:45.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:44.526390+0000 mon.a (mon.0) 396 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:45.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:44.531374+0000 mon.a (mon.0) 397 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:45.092 DEBUG:teuthology.orchestra.run.smithi179:osd.6> sudo journalctl -f -n 0 -u ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.6.service 2024-09-17T07:11:45.094 INFO:tasks.cephadm:Deploying osd.7 on smithi179 with /dev/vg_nvme/lv_1... 2024-09-17T07:11:45.095 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-09-17T07:11:46.955 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:45.759561+0000 mgr.y (mgr.14144) 129 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:11:47.071 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-09-17T07:11:47.071 INFO:teuthology.orchestra.run.smithi179.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-09-17T07:11:47.071 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10+0 records in 2024-09-17T07:11:47.071 INFO:teuthology.orchestra.run.smithi179.stderr:10+0 records out 2024-09-17T07:11:47.071 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126774 s, 827 MB/s 2024-09-17T07:11:47.071 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping successful for: 2024-09-17T07:11:47.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:45.759561+0000 mgr.y (mgr.14144) 129 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:11:47.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:45.759561+0000 mgr.y (mgr.14144) 129 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:11:47.900 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:47.648894+0000 mon.b (mon.2) 16 : audit [INF] from='osd.6 [v2:172.21.15.179:6816/1852137393,v1:172.21.15.179:6817/1852137393]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T07:11:47.900 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:47.649303+0000 mon.a (mon.0) 398 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T07:11:47.900 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:11:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.6[31384]: debug 2024-09-17T07:11:47.645+0000 7f1e71857080 -1 osd.6 0 log_to_monitors {default=true} 2024-09-17T07:11:47.922 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi179:vg_nvme/lv_1 2024-09-17T07:11:48.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:47.648894+0000 mon.b (mon.2) 16 : audit [INF] from='osd.6 [v2:172.21.15.179:6816/1852137393,v1:172.21.15.179:6817/1852137393]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T07:11:48.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:47.649303+0000 mon.a (mon.0) 398 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T07:11:48.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:47.648894+0000 mon.b (mon.2) 16 : audit [INF] from='osd.6 [v2:172.21.15.179:6816/1852137393,v1:172.21.15.179:6817/1852137393]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T07:11:48.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:47.649303+0000 mon.a (mon.0) 398 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T07:11:49.114 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:11:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.6[31384]: debug 2024-09-17T07:11:48.834+0000 7f1e67dc8700 -1 osd.6 0 waiting for initial osdmap 2024-09-17T07:11:49.114 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:11:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.6[31384]: debug 2024-09-17T07:11:48.841+0000 7f1e62540700 -1 osd.6 41 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2024-09-17T07:11:49.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:47.759987+0000 mgr.y (mgr.14144) 130 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:11:49.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:47.824236+0000 mon.a (mon.0) 399 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-17T07:11:49.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:47.824335+0000 mon.a (mon.0) 400 : cluster [DBG] osdmap e40: 7 total, 6 up, 7 in 2024-09-17T07:11:49.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:47.824493+0000 mon.a (mon.0) 401 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:11:49.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:47.825737+0000 mon.b (mon.2) 17 : audit [INF] from='osd.6 [v2:172.21.15.179:6816/1852137393,v1:172.21.15.179:6817/1852137393]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:11:49.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:47.826127+0000 mon.a (mon.0) 402 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:11:49.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:47.865765+0000 mon.a (mon.0) 403 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:11:49.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:47.866815+0000 mon.a (mon.0) 404 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:11:49.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:48.073120+0000 mon.a (mon.0) 405 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:49.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:47.759987+0000 mgr.y (mgr.14144) 130 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:11:49.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:47.824236+0000 mon.a (mon.0) 399 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-17T07:11:49.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:47.824335+0000 mon.a (mon.0) 400 : cluster [DBG] osdmap e40: 7 total, 6 up, 7 in 2024-09-17T07:11:49.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:47.824493+0000 mon.a (mon.0) 401 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:11:49.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:47.825737+0000 mon.b (mon.2) 17 : audit [INF] from='osd.6 [v2:172.21.15.179:6816/1852137393,v1:172.21.15.179:6817/1852137393]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:11:49.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:47.826127+0000 mon.a (mon.0) 402 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:11:49.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:47.865765+0000 mon.a (mon.0) 403 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:11:49.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:47.866815+0000 mon.a (mon.0) 404 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:11:49.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:48.073120+0000 mon.a (mon.0) 405 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:49.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:47.759987+0000 mgr.y (mgr.14144) 130 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:11:49.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:47.824236+0000 mon.a (mon.0) 399 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-17T07:11:49.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:47.824335+0000 mon.a (mon.0) 400 : cluster [DBG] osdmap e40: 7 total, 6 up, 7 in 2024-09-17T07:11:49.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:47.824493+0000 mon.a (mon.0) 401 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:11:49.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:47.825737+0000 mon.b (mon.2) 17 : audit [INF] from='osd.6 [v2:172.21.15.179:6816/1852137393,v1:172.21.15.179:6817/1852137393]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:11:49.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:47.826127+0000 mon.a (mon.0) 402 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:11:49.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:47.865765+0000 mon.a (mon.0) 403 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:11:49.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:47.866815+0000 mon.a (mon.0) 404 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:11:49.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:48.073120+0000 mon.a (mon.0) 405 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:50.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:48.828598+0000 mon.a (mon.0) 406 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-09-17T07:11:50.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:48.828784+0000 mon.a (mon.0) 407 : cluster [DBG] osdmap e41: 7 total, 6 up, 7 in 2024-09-17T07:11:50.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:48.829397+0000 mon.a (mon.0) 408 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:11:50.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:48.832262+0000 mon.a (mon.0) 409 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:11:50.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:49.624917+0000 mon.a (mon.0) 410 : cluster [INF] osd.6 [v2:172.21.15.179:6816/1852137393,v1:172.21.15.179:6817/1852137393] boot 2024-09-17T07:11:50.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:49.624998+0000 mon.a (mon.0) 411 : cluster [DBG] osdmap e42: 7 total, 7 up, 7 in 2024-09-17T07:11:50.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:49.625547+0000 mon.a (mon.0) 412 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:11:50.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:48.828598+0000 mon.a (mon.0) 406 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-09-17T07:11:50.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:48.828784+0000 mon.a (mon.0) 407 : cluster [DBG] osdmap e41: 7 total, 6 up, 7 in 2024-09-17T07:11:50.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:48.829397+0000 mon.a (mon.0) 408 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:11:50.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:48.832262+0000 mon.a (mon.0) 409 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:11:50.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:49.624917+0000 mon.a (mon.0) 410 : cluster [INF] osd.6 [v2:172.21.15.179:6816/1852137393,v1:172.21.15.179:6817/1852137393] boot 2024-09-17T07:11:50.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:49.624998+0000 mon.a (mon.0) 411 : cluster [DBG] osdmap e42: 7 total, 7 up, 7 in 2024-09-17T07:11:50.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:49.625547+0000 mon.a (mon.0) 412 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:11:50.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:48.828598+0000 mon.a (mon.0) 406 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-09-17T07:11:50.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:48.828784+0000 mon.a (mon.0) 407 : cluster [DBG] osdmap e41: 7 total, 6 up, 7 in 2024-09-17T07:11:50.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:48.829397+0000 mon.a (mon.0) 408 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:11:50.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:48.832262+0000 mon.a (mon.0) 409 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:11:50.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:49.624917+0000 mon.a (mon.0) 410 : cluster [INF] osd.6 [v2:172.21.15.179:6816/1852137393,v1:172.21.15.179:6817/1852137393] boot 2024-09-17T07:11:50.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:49.624998+0000 mon.a (mon.0) 411 : cluster [DBG] osdmap e42: 7 total, 7 up, 7 in 2024-09-17T07:11:50.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:49.625547+0000 mon.a (mon.0) 412 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:11:51.096 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:48.674804+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T07:11:51.096 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:48.674900+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T07:11:51.097 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:49.760424+0000 mgr.y (mgr.14144) 131 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:11:51.097 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:50.637614+0000 mon.a (mon.0) 413 : cluster [DBG] osdmap e43: 7 total, 7 up, 7 in 2024-09-17T07:11:51.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:48.674804+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T07:11:51.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:48.674900+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T07:11:51.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:49.760424+0000 mgr.y (mgr.14144) 131 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:11:51.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:50.637614+0000 mon.a (mon.0) 413 : cluster [DBG] osdmap e43: 7 total, 7 up, 7 in 2024-09-17T07:11:51.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:48.674804+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T07:11:51.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:48.674900+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T07:11:51.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:49.760424+0000 mgr.y (mgr.14144) 131 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2024-09-17T07:11:51.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:50.637614+0000 mon.a (mon.0) 413 : cluster [DBG] osdmap e43: 7 total, 7 up, 7 in 2024-09-17T07:11:52.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:51.063611+0000 mgr.y (mgr.14144) 132 : audit [DBG] from='client.34268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:11:52.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:51.064531+0000 mon.a (mon.0) 414 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:11:52.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:11:51.065934+0000 mgr.y (mgr.14144) 133 : cephadm [INF] Applying service osd.None on host smithi179... 2024-09-17T07:11:52.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:51.066230+0000 mon.a (mon.0) 415 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:11:52.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:51.066972+0000 mon.a (mon.0) 416 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:11:52.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:51.637544+0000 mon.a (mon.0) 417 : cluster [DBG] osdmap e44: 7 total, 7 up, 7 in 2024-09-17T07:11:52.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:51.063611+0000 mgr.y (mgr.14144) 132 : audit [DBG] from='client.34268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:11:52.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:51.064531+0000 mon.a (mon.0) 414 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:11:52.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:11:51.065934+0000 mgr.y (mgr.14144) 133 : cephadm [INF] Applying service osd.None on host smithi179... 2024-09-17T07:11:52.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:51.066230+0000 mon.a (mon.0) 415 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:11:52.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:51.066972+0000 mon.a (mon.0) 416 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:11:52.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:51.637544+0000 mon.a (mon.0) 417 : cluster [DBG] osdmap e44: 7 total, 7 up, 7 in 2024-09-17T07:11:52.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:51.063611+0000 mgr.y (mgr.14144) 132 : audit [DBG] from='client.34268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:11:52.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:51.064531+0000 mon.a (mon.0) 414 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T07:11:52.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:11:51.065934+0000 mgr.y (mgr.14144) 133 : cephadm [INF] Applying service osd.None on host smithi179... 2024-09-17T07:11:52.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:51.066230+0000 mon.a (mon.0) 415 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T07:11:52.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:51.066972+0000 mon.a (mon.0) 416 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:11:52.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:51.637544+0000 mon.a (mon.0) 417 : cluster [DBG] osdmap e44: 7 total, 7 up, 7 in 2024-09-17T07:11:52.977 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:51.760800+0000 mgr.y (mgr.14144) 134 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:11:53.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:51.760800+0000 mgr.y (mgr.14144) 134 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:11:53.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:51.760800+0000 mgr.y (mgr.14144) 134 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:11:54.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:11:53.189716+0000 mgr.y (mgr.14144) 135 : cephadm [INF] Detected new or changed devices on smithi179 2024-09-17T07:11:54.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:53.194511+0000 mon.a (mon.0) 418 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:54.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:53.198500+0000 mon.a (mon.0) 419 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:54.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:53.200107+0000 mon.a (mon.0) 420 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:11:54.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:53.204823+0000 mon.a (mon.0) 421 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:54.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:54.000527+0000 mon.b (mon.2) 18 : audit [INF] from='client.? 172.21.15.179:0/783491421' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1cfecfc-fb47-4667-939a-adb55b684468"}]: dispatch 2024-09-17T07:11:54.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:54.001196+0000 mon.a (mon.0) 422 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1cfecfc-fb47-4667-939a-adb55b684468"}]: dispatch 2024-09-17T07:11:54.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:54.007040+0000 mon.a (mon.0) 423 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d1cfecfc-fb47-4667-939a-adb55b684468"}]': finished 2024-09-17T07:11:54.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:54.007151+0000 mon.a (mon.0) 424 : cluster [DBG] osdmap e45: 8 total, 7 up, 8 in 2024-09-17T07:11:54.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:54.007326+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:11:54.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:11:53.189716+0000 mgr.y (mgr.14144) 135 : cephadm [INF] Detected new or changed devices on smithi179 2024-09-17T07:11:54.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:53.194511+0000 mon.a (mon.0) 418 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:54.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:53.198500+0000 mon.a (mon.0) 419 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:54.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:53.200107+0000 mon.a (mon.0) 420 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:11:54.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:53.204823+0000 mon.a (mon.0) 421 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:54.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:54.000527+0000 mon.b (mon.2) 18 : audit [INF] from='client.? 172.21.15.179:0/783491421' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1cfecfc-fb47-4667-939a-adb55b684468"}]: dispatch 2024-09-17T07:11:54.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:54.001196+0000 mon.a (mon.0) 422 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1cfecfc-fb47-4667-939a-adb55b684468"}]: dispatch 2024-09-17T07:11:54.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:54.007040+0000 mon.a (mon.0) 423 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d1cfecfc-fb47-4667-939a-adb55b684468"}]': finished 2024-09-17T07:11:54.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:54.007151+0000 mon.a (mon.0) 424 : cluster [DBG] osdmap e45: 8 total, 7 up, 8 in 2024-09-17T07:11:54.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:54.007326+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:11:54.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:11:53.189716+0000 mgr.y (mgr.14144) 135 : cephadm [INF] Detected new or changed devices on smithi179 2024-09-17T07:11:54.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:53.194511+0000 mon.a (mon.0) 418 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:54.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:53.198500+0000 mon.a (mon.0) 419 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:54.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:53.200107+0000 mon.a (mon.0) 420 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:11:54.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:53.204823+0000 mon.a (mon.0) 421 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:11:54.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:54.000527+0000 mon.b (mon.2) 18 : audit [INF] from='client.? 172.21.15.179:0/783491421' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1cfecfc-fb47-4667-939a-adb55b684468"}]: dispatch 2024-09-17T07:11:54.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:54.001196+0000 mon.a (mon.0) 422 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1cfecfc-fb47-4667-939a-adb55b684468"}]: dispatch 2024-09-17T07:11:54.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:54.007040+0000 mon.a (mon.0) 423 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d1cfecfc-fb47-4667-939a-adb55b684468"}]': finished 2024-09-17T07:11:54.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:54.007151+0000 mon.a (mon.0) 424 : cluster [DBG] osdmap e45: 8 total, 7 up, 8 in 2024-09-17T07:11:54.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:54.007326+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:11:55.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:53.761224+0000 mgr.y (mgr.14144) 136 : cluster [DBG] pgmap v127: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:11:55.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:54.450576+0000 mon.b (mon.2) 19 : audit [DBG] from='client.? 172.21.15.179:0/115402034' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:11:55.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:53.761224+0000 mgr.y (mgr.14144) 136 : cluster [DBG] pgmap v127: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:11:55.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:54.450576+0000 mon.b (mon.2) 19 : audit [DBG] from='client.? 172.21.15.179:0/115402034' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:11:55.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:53.761224+0000 mgr.y (mgr.14144) 136 : cluster [DBG] pgmap v127: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:11:55.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:54.450576+0000 mon.b (mon.2) 19 : audit [DBG] from='client.? 172.21.15.179:0/115402034' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T07:11:57.293 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:55.761540+0000 mgr.y (mgr.14144) 137 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:11:57.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:55.761540+0000 mgr.y (mgr.14144) 137 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:11:57.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:55.761540+0000 mgr.y (mgr.14144) 137 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:11:59.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:11:59 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:57.761927+0000 mgr.y (mgr.14144) 138 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:11:59.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:11:59 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:57.761927+0000 mgr.y (mgr.14144) 138 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:11:59.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:11:59 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:57.761927+0000 mgr.y (mgr.14144) 138 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:12:00.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:00 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:11:59.762401+0000 mgr.y (mgr.14144) 139 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:12:00.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:00 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:59.972872+0000 mon.a (mon.0) 426 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-17T07:12:00.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:00 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:11:59.973680+0000 mon.a (mon.0) 427 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:12:00.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:00 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:11:59.974364+0000 mgr.y (mgr.14144) 140 : cephadm [INF] Deploying daemon osd.7 on smithi179 2024-09-17T07:12:01.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:11:59.762401+0000 mgr.y (mgr.14144) 139 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:12:01.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:59.972872+0000 mon.a (mon.0) 426 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-17T07:12:01.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:11:59.973680+0000 mon.a (mon.0) 427 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:12:01.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:11:59.974364+0000 mgr.y (mgr.14144) 140 : cephadm [INF] Deploying daemon osd.7 on smithi179 2024-09-17T07:12:01.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:11:59.762401+0000 mgr.y (mgr.14144) 139 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:12:01.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:59.972872+0000 mon.a (mon.0) 426 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-17T07:12:01.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:11:59.973680+0000 mon.a (mon.0) 427 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:12:01.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:11:59.974364+0000 mgr.y (mgr.14144) 140 : cephadm [INF] Deploying daemon osd.7 on smithi179 2024-09-17T07:12:03.064 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:01.762787+0000 mgr.y (mgr.14144) 141 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:12:03.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:01.762787+0000 mgr.y (mgr.14144) 141 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:12:03.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:01.762787+0000 mgr.y (mgr.14144) 141 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:12:03.483 INFO:teuthology.orchestra.run.smithi179.stdout:Created osd(s) 7 on host 'smithi179' 2024-09-17T07:12:04.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:03 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:03.471726+0000 mon.a (mon.0) 428 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:12:04.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:03 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:03.476167+0000 mon.a (mon.0) 429 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:04.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:03 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:03.480835+0000 mon.a (mon.0) 430 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:04.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:03 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:03.471726+0000 mon.a (mon.0) 428 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:12:04.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:03 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:03.476167+0000 mon.a (mon.0) 429 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:04.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:03 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:03.480835+0000 mon.a (mon.0) 430 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:04.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:03 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:03.471726+0000 mon.a (mon.0) 428 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:12:04.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:03 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:03.476167+0000 mon.a (mon.0) 429 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:04.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:03 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:03.480835+0000 mon.a (mon.0) 430 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:04.205 DEBUG:teuthology.orchestra.run.smithi179:osd.7> sudo journalctl -f -n 0 -u ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.7.service 2024-09-17T07:12:04.208 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-17T07:12:04.208 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-17T07:12:05.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:03.763238+0000 mgr.y (mgr.14144) 142 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:12:05.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:05.167334+0000 mon.a (mon.0) 431 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:05.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:03.763238+0000 mgr.y (mgr.14144) 142 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:12:05.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:05.167334+0000 mon.a (mon.0) 431 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:05.773 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:05 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:03.763238+0000 mgr.y (mgr.14144) 142 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:12:05.773 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:05 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:05.167334+0000 mon.a (mon.0) 431 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:06.431 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-09-17T07:12:06.646 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:06.430832+0000 mon.a (mon.0) 432 : audit [DBG] from='client.? 172.21.15.44:0/1855387241' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-17T07:12:06.647 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:12:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.7[33776]: debug 2024-09-17T07:12:06.547+0000 7fa540104080 -1 osd.7 0 log_to_monitors {default=true} 2024-09-17T07:12:06.788 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:06.430832+0000 mon.a (mon.0) 432 : audit [DBG] from='client.? 172.21.15.44:0/1855387241' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-17T07:12:06.788 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:06.430832+0000 mon.a (mon.0) 432 : audit [DBG] from='client.? 172.21.15.44:0/1855387241' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-17T07:12:07.065 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":45,"num_osds":8,"num_up_osds":7,"osd_up_since":1726557109,"num_in_osds":8,"osd_in_since":1726557114,"num_remapped_pgs":0} 2024-09-17T07:12:07.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:05.763590+0000 mgr.y (mgr.14144) 143 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:12:07.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:06.551405+0000 mon.b (mon.2) 20 : audit [INF] from='osd.7 [v2:172.21.15.179:6824/191862342,v1:172.21.15.179:6825/191862342]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T07:12:07.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:06.551823+0000 mon.a (mon.0) 433 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T07:12:07.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:05.763590+0000 mgr.y (mgr.14144) 143 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:12:07.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:06.551405+0000 mon.b (mon.2) 20 : audit [INF] from='osd.7 [v2:172.21.15.179:6824/191862342,v1:172.21.15.179:6825/191862342]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T07:12:07.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:06.551823+0000 mon.a (mon.0) 433 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T07:12:07.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:05.763590+0000 mgr.y (mgr.14144) 143 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:12:07.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:06.551405+0000 mon.b (mon.2) 20 : audit [INF] from='osd.7 [v2:172.21.15.179:6824/191862342,v1:172.21.15.179:6825/191862342]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T07:12:07.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:06.551823+0000 mon.a (mon.0) 433 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T07:12:08.066 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-17T07:12:08.778 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:07.500258+0000 mon.a (mon.0) 434 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-17T07:12:08.778 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:07.500342+0000 mon.a (mon.0) 435 : cluster [DBG] osdmap e46: 8 total, 7 up, 8 in 2024-09-17T07:12:08.778 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:07.500461+0000 mon.a (mon.0) 436 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:12:08.778 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:07.501030+0000 mon.b (mon.2) 21 : audit [INF] from='osd.7 [v2:172.21.15.179:6824/191862342,v1:172.21.15.179:6825/191862342]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:12:08.779 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:07.501323+0000 mon.a (mon.0) 437 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:12:08.779 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:08.407425+0000 mon.a (mon.0) 438 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:08.779 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:08.410480+0000 mon.a (mon.0) 439 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:08.779 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:08.411935+0000 mon.a (mon.0) 440 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:12:08.779 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:07.500258+0000 mon.a (mon.0) 434 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-17T07:12:08.779 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:07.500342+0000 mon.a (mon.0) 435 : cluster [DBG] osdmap e46: 8 total, 7 up, 8 in 2024-09-17T07:12:08.779 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:07.500461+0000 mon.a (mon.0) 436 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:12:08.779 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:07.501030+0000 mon.b (mon.2) 21 : audit [INF] from='osd.7 [v2:172.21.15.179:6824/191862342,v1:172.21.15.179:6825/191862342]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:12:08.779 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:07.501323+0000 mon.a (mon.0) 437 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:12:08.779 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:08.407425+0000 mon.a (mon.0) 438 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:08.779 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:08.410480+0000 mon.a (mon.0) 439 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:08.780 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:08.411935+0000 mon.a (mon.0) 440 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:12:08.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:12:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.7[33776]: debug 2024-09-17T07:12:08.512+0000 7fa536675700 -1 osd.7 0 waiting for initial osdmap 2024-09-17T07:12:08.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:12:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.7[33776]: debug 2024-09-17T07:12:08.520+0000 7fa531def700 -1 osd.7 47 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2024-09-17T07:12:08.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:07.500258+0000 mon.a (mon.0) 434 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-17T07:12:08.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:07.500342+0000 mon.a (mon.0) 435 : cluster [DBG] osdmap e46: 8 total, 7 up, 8 in 2024-09-17T07:12:08.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:07.500461+0000 mon.a (mon.0) 436 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:12:08.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:07.501030+0000 mon.b (mon.2) 21 : audit [INF] from='osd.7 [v2:172.21.15.179:6824/191862342,v1:172.21.15.179:6825/191862342]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:12:08.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:07.501323+0000 mon.a (mon.0) 437 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:12:08.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:08.407425+0000 mon.a (mon.0) 438 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:08.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:08.410480+0000 mon.a (mon.0) 439 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:08.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:08.411935+0000 mon.a (mon.0) 440 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:12:09.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:07.763957+0000 mgr.y (mgr.14144) 144 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:12:09.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:12:08.402842+0000 mgr.y (mgr.14144) 145 : cephadm [INF] Detected new or changed devices on smithi179 2024-09-17T07:12:09.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:08.504768+0000 mon.a (mon.0) 441 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-09-17T07:12:09.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:08.504862+0000 mon.a (mon.0) 442 : cluster [DBG] osdmap e47: 8 total, 7 up, 8 in 2024-09-17T07:12:09.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:08.505331+0000 mon.a (mon.0) 443 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:12:09.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:08.508800+0000 mon.a (mon.0) 444 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:12:09.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:08.637120+0000 mon.a (mon.0) 445 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:09.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:09.509008+0000 mon.a (mon.0) 446 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:12:09.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:07.763957+0000 mgr.y (mgr.14144) 144 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:12:09.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:12:08.402842+0000 mgr.y (mgr.14144) 145 : cephadm [INF] Detected new or changed devices on smithi179 2024-09-17T07:12:09.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:08.504768+0000 mon.a (mon.0) 441 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-09-17T07:12:09.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:08.504862+0000 mon.a (mon.0) 442 : cluster [DBG] osdmap e47: 8 total, 7 up, 8 in 2024-09-17T07:12:09.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:08.505331+0000 mon.a (mon.0) 443 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:12:09.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:08.508800+0000 mon.a (mon.0) 444 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:12:09.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:08.637120+0000 mon.a (mon.0) 445 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:09.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:09.509008+0000 mon.a (mon.0) 446 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:12:09.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:07.763957+0000 mgr.y (mgr.14144) 144 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:12:09.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:12:08.402842+0000 mgr.y (mgr.14144) 145 : cephadm [INF] Detected new or changed devices on smithi179 2024-09-17T07:12:09.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:08.504768+0000 mon.a (mon.0) 441 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-09-17T07:12:09.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:08.504862+0000 mon.a (mon.0) 442 : cluster [DBG] osdmap e47: 8 total, 7 up, 8 in 2024-09-17T07:12:09.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:08.505331+0000 mon.a (mon.0) 443 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:12:09.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:08.508800+0000 mon.a (mon.0) 444 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:12:09.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:08.637120+0000 mon.a (mon.0) 445 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:09.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:09.509008+0000 mon.a (mon.0) 446 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:12:10.213 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-09-17T07:12:10.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:07.566594+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T07:12:10.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:07.566770+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T07:12:10.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:09.518067+0000 mon.a (mon.0) 447 : cluster [INF] osd.7 [v2:172.21.15.179:6824/191862342,v1:172.21.15.179:6825/191862342] boot 2024-09-17T07:12:10.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:09.518098+0000 mon.a (mon.0) 448 : cluster [DBG] osdmap e48: 8 total, 8 up, 8 in 2024-09-17T07:12:10.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:09.518216+0000 mon.a (mon.0) 449 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:12:10.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:09.627006+0000 mon.a (mon.0) 450 : cluster [DBG] osdmap e49: 8 total, 8 up, 8 in 2024-09-17T07:12:10.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:10.212688+0000 mon.c (mon.1) 8 : audit [DBG] from='client.? 172.21.15.44:0/3883351042' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-17T07:12:10.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:07.566594+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T07:12:10.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:07.566770+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T07:12:10.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:09.518067+0000 mon.a (mon.0) 447 : cluster [INF] osd.7 [v2:172.21.15.179:6824/191862342,v1:172.21.15.179:6825/191862342] boot 2024-09-17T07:12:10.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:09.518098+0000 mon.a (mon.0) 448 : cluster [DBG] osdmap e48: 8 total, 8 up, 8 in 2024-09-17T07:12:10.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:09.518216+0000 mon.a (mon.0) 449 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:12:10.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:09.627006+0000 mon.a (mon.0) 450 : cluster [DBG] osdmap e49: 8 total, 8 up, 8 in 2024-09-17T07:12:10.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:10.212688+0000 mon.c (mon.1) 8 : audit [DBG] from='client.? 172.21.15.44:0/3883351042' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-17T07:12:10.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:07.566594+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T07:12:10.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:07.566770+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T07:12:10.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:09.518067+0000 mon.a (mon.0) 447 : cluster [INF] osd.7 [v2:172.21.15.179:6824/191862342,v1:172.21.15.179:6825/191862342] boot 2024-09-17T07:12:10.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:09.518098+0000 mon.a (mon.0) 448 : cluster [DBG] osdmap e48: 8 total, 8 up, 8 in 2024-09-17T07:12:10.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:09.518216+0000 mon.a (mon.0) 449 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:12:10.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:09.627006+0000 mon.a (mon.0) 450 : cluster [DBG] osdmap e49: 8 total, 8 up, 8 in 2024-09-17T07:12:10.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:10.212688+0000 mon.c (mon.1) 8 : audit [DBG] from='client.? 172.21.15.44:0/3883351042' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-17T07:12:10.917 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":49,"num_osds":8,"num_up_osds":8,"osd_up_since":1726557129,"num_in_osds":8,"osd_in_since":1726557114,"num_remapped_pgs":0} 2024-09-17T07:12:10.918 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-17T07:12:11.126 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/817ae996-74c3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:12:11.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:09.764226+0000 mgr.y (mgr.14144) 146 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:12:11.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:10.638110+0000 mon.a (mon.0) 451 : cluster [DBG] osdmap e50: 8 total, 8 up, 8 in 2024-09-17T07:12:11.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:09.764226+0000 mgr.y (mgr.14144) 146 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:12:11.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:10.638110+0000 mon.a (mon.0) 451 : cluster [DBG] osdmap e50: 8 total, 8 up, 8 in 2024-09-17T07:12:11.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:11 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:09.764226+0000 mgr.y (mgr.14144) 146 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T07:12:11.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:11 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:10.638110+0000 mon.a (mon.0) 451 : cluster [DBG] osdmap e50: 8 total, 8 up, 8 in 2024-09-17T07:12:12.889 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-09-17T07:12:12.889 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":50,"fsid":"817ae996-74c3-11ef-bceb-c7b262605968","created":"2024-09-17T07:08:04.408168+0000","modified":"2024-09-17T07:12:10.632496+0000","last_up_change":"2024-09-17T07:12:09.510313+0000","last_in_change":"2024-09-17T07:11:54.001817+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-09-17T07:10:35.776015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"7f236893-2f40-4119-ab88-7359969cc30a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":1979937622},{"type":"v1","addr":"172.21.15.44:6803","nonce":1979937622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":1979937622},{"type":"v1","addr":"172.21.15.44:6805","nonce":1979937622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":1979937622},{"type":"v1","addr":"172.21.15.44:6809","nonce":1979937622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":1979937622},{"type":"v1","addr":"172.21.15.44:6807","nonce":1979937622}]},"public_addr":"172.21.15.44:6803/1979937622","cluster_addr":"172.21.15.44:6805/1979937622","heartbeat_back_addr":"172.21.15.44:6809/1979937622","heartbeat_front_addr":"172.21.15.44:6807/1979937622","state":["exists","up"]},{"osd":1,"uuid":"f8da14c0-ab31-42e2-be11-bd734d990923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":1594621847},{"type":"v1","addr":"172.21.15.44:6811","nonce":1594621847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":1594621847},{"type":"v1","addr":"172.21.15.44:6813","nonce":1594621847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":1594621847},{"type":"v1","addr":"172.21.15.44:6817","nonce":1594621847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":1594621847},{"type":"v1","addr":"172.21.15.44:6815","nonce":1594621847}]},"public_addr":"172.21.15.44:6811/1594621847","cluster_addr":"172.21.15.44:6813/1594621847","heartbeat_back_addr":"172.21.15.44:6817/1594621847","heartbeat_front_addr":"172.21.15.44:6815/1594621847","state":["exists","up"]},{"osd":2,"uuid":"68dff20f-443b-4ed3-86a5-dc84cb5b9da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":3871662332},{"type":"v1","addr":"172.21.15.44:6819","nonce":3871662332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":3871662332},{"type":"v1","addr":"172.21.15.44:6821","nonce":3871662332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":3871662332},{"type":"v1","addr":"172.21.15.44:6825","nonce":3871662332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":3871662332},{"type":"v1","addr":"172.21.15.44:6823","nonce":3871662332}]},"public_addr":"172.21.15.44:6819/3871662332","cluster_addr":"172.21.15.44:6821/3871662332","heartbeat_back_addr":"172.21.15.44:6825/3871662332","heartbeat_front_addr":"172.21.15.44:6823/3871662332","state":["exists","up"]},{"osd":3,"uuid":"72f5c16b-cdf4-45fa-a49b-93fed3cf8d07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":905450488},{"type":"v1","addr":"172.21.15.44:6827","nonce":905450488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":905450488},{"type":"v1","addr":"172.21.15.44:6829","nonce":905450488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":905450488},{"type":"v1","addr":"172.21.15.44:6833","nonce":905450488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":905450488},{"type":"v1","addr":"172.21.15.44:6831","nonce":905450488}]},"public_addr":"172.21.15.44:6827/905450488","cluster_addr":"172.21.15.44:6829/905450488","heartbeat_back_addr":"172.21.15.44:6833/905450488","heartbeat_front_addr":"172.21.15.44:6831/905450488","state":["exists","up"]},{"osd":4,"uuid":"ff638478-3d19-4b13-9123-4be3ad21a531","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2112811328},{"type":"v1","addr":"172.21.15.179:6801","nonce":2112811328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2112811328},{"type":"v1","addr":"172.21.15.179:6803","nonce":2112811328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2112811328},{"type":"v1","addr":"172.21.15.179:6807","nonce":2112811328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2112811328},{"type":"v1","addr":"172.21.15.179:6805","nonce":2112811328}]},"public_addr":"172.21.15.179:6801/2112811328","cluster_addr":"172.21.15.179:6803/2112811328","heartbeat_back_addr":"172.21.15.179:6807/2112811328","heartbeat_front_addr":"172.21.15.179:6805/2112811328","state":["exists","up"]},{"osd":5,"uuid":"83cd743c-9dcb-4027-8857-34773603b77c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":3243120327},{"type":"v1","addr":"172.21.15.179:6809","nonce":3243120327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":3243120327},{"type":"v1","addr":"172.21.15.179:6811","nonce":3243120327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":3243120327},{"type":"v1","addr":"172.21.15.179:6815","nonce":3243120327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":3243120327},{"type":"v1","addr":"172.21.15.179:6813","nonce":3243120327}]},"public_addr":"172.21.15.179:6809/3243120327","cluster_addr":"172.21.15.179:6811/3243120327","heartbeat_back_addr":"172.21.15.179:6815/3243120327","heartbeat_front_addr":"172.21.15.179:6813/3243120327","state":["exists","up"]},{"osd":6,"uuid":"4e608f05-0fef-407f-aa40-ded55acd0bb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":1852137393},{"type":"v1","addr":"172.21.15.179:6817","nonce":1852137393}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":1852137393},{"type":"v1","addr":"172.21.15.179:6819","nonce":1852137393}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":1852137393},{"type":"v1","addr":"172.21.15.179:6823","nonce":1852137393}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":1852137393},{"type":"v1","addr":"172.21.15.179:6821","nonce":1852137393}]},"public_addr":"172.21.15.179:6817/1852137393","cluster_addr":"172.21.15.179:6819/1852137393","heartbeat_back_addr":"172.21.15.179:6823/1852137393","heartbeat_front_addr":"172.21.15.179:6821/1852137393","state":["exists","up"]},{"osd":7,"uuid":"d1cfecfc-fb47-4667-939a-adb55b684468","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":191862342},{"type":"v1","addr":"172.21.15.179:6825","nonce":191862342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":191862342},{"type":"v1","addr":"172.21.15.179:6827","nonce":191862342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":191862342},{"type":"v1","addr":"172.21.15.179:6831","nonce":191862342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":191862342},{"type":"v1","addr":"172.21.15.179:6829","nonce":191862342}]},"public_addr":"172.21.15.179:6825/191862342","cluster_addr":"172.21.15.179:6827/191862342","heartbeat_back_addr":"172.21.15.179:6831/191862342","heartbeat_front_addr":"172.21.15.179:6829/191862342","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:09:55.262575+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:10:13.977151+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:10:33.368706+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:10:53.567070+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:11:09.924018+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:11:29.294199+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:11:48.674903+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:12:07.566776+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/3961128045":"2024-09-18T07:08:47.676584+0000","172.21.15.44:0/367718499":"2024-09-18T07:08:47.676584+0000","172.21.15.44:6800/930976792":"2024-09-18T07:08:47.676584+0000","172.21.15.44:6801/930976792":"2024-09-18T07:08:47.676584+0000","172.21.15.44:0/1289983812":"2024-09-18T07:08:26.744189+0000","172.21.15.44:0/300796373":"2024-09-18T07:08:26.744189+0000","172.21.15.44:6801/2448217567":"2024-09-18T07:08:26.744189+0000","172.21.15.44:6800/2448217567":"2024-09-18T07:08:26.744189+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T07:12:13.497 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-09-17T07:10:35.776015+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}] 2024-09-17T07:12:13.498 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph osd pool get device_health_metrics pg_num 2024-09-17T07:12:13.702 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/817ae996-74c3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:12:13.823 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:11.764643+0000 mgr.y (mgr.14144) 147 : cluster [DBG] pgmap v142: 1 pgs: 1 peering; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:13.823 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:12.889046+0000 mon.c (mon.1) 9 : audit [DBG] from='client.? 172.21.15.44:0/3251783271' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T07:12:13.823 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:11.764643+0000 mgr.y (mgr.14144) 147 : cluster [DBG] pgmap v142: 1 pgs: 1 peering; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:13.823 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:12.889046+0000 mon.c (mon.1) 9 : audit [DBG] from='client.? 172.21.15.44:0/3251783271' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T07:12:13.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:13 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:11.764643+0000 mgr.y (mgr.14144) 147 : cluster [DBG] pgmap v142: 1 pgs: 1 peering; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:13.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:13 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:12.889046+0000 mon.c (mon.1) 9 : audit [DBG] from='client.? 172.21.15.44:0/3251783271' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T07:12:15.455 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-09-17T07:12:15.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:13.764983+0000 mgr.y (mgr.14144) 148 : cluster [DBG] pgmap v143: 1 pgs: 1 peering; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:15.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:15.454347+0000 mon.a (mon.0) 452 : audit [DBG] from='client.? 172.21.15.44:0/3867907092' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2024-09-17T07:12:15.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:13.764983+0000 mgr.y (mgr.14144) 148 : cluster [DBG] pgmap v143: 1 pgs: 1 peering; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:15.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:15.454347+0000 mon.a (mon.0) 452 : audit [DBG] from='client.? 172.21.15.44:0/3867907092' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2024-09-17T07:12:15.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:13.764983+0000 mgr.y (mgr.14144) 148 : cluster [DBG] pgmap v143: 1 pgs: 1 peering; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:15.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:15.454347+0000 mon.a (mon.0) 452 : audit [DBG] from='client.? 172.21.15.44:0/3867907092' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2024-09-17T07:12:16.129 INFO:tasks.cephadm:Adding prometheus.a on smithi179 2024-09-17T07:12:16.129 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph orch apply prometheus '1;smithi179=a' 2024-09-17T07:12:17.673 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:15.765328+0000 mgr.y (mgr.14144) 149 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:17.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:17 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:15.765328+0000 mgr.y (mgr.14144) 149 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:17.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:17 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:15.765328+0000 mgr.y (mgr.14144) 149 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:18.080 INFO:teuthology.orchestra.run.smithi179.stdout:Scheduled prometheus update... 2024-09-17T07:12:18.724 DEBUG:teuthology.orchestra.run.smithi179:prometheus.a> sudo journalctl -f -n 0 -u ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service 2024-09-17T07:12:18.726 INFO:tasks.cephadm:Adding node-exporter.a on smithi044 2024-09-17T07:12:18.726 INFO:tasks.cephadm:Adding node-exporter.b on smithi179 2024-09-17T07:12:18.726 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph orch apply node-exporter '2;smithi044=a;smithi179=b' 2024-09-17T07:12:19.346 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:19 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:17.765799+0000 mgr.y (mgr.14144) 150 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:19.346 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:19 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:18.072431+0000 mgr.y (mgr.14144) 151 : audit [DBG] from='client.34313 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi179=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:12:19.347 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:19 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:12:18.073193+0000 mgr.y (mgr.14144) 152 : cephadm [INF] Saving service prometheus spec with placement smithi179=a;count:1 2024-09-17T07:12:19.347 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:19 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:18.077851+0000 mon.a (mon.0) 453 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:19.347 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:19 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:18.078684+0000 mon.a (mon.0) 454 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:12:19.347 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:19 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:18.295415+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:19.347 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:19 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:12:18.306687+0000 mgr.y (mgr.14144) 153 : cephadm [INF] Deploying daemon prometheus.a on smithi179 2024-09-17T07:12:19.574 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:17.765799+0000 mgr.y (mgr.14144) 150 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:19.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:18.072431+0000 mgr.y (mgr.14144) 151 : audit [DBG] from='client.34313 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi179=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:12:19.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:12:18.073193+0000 mgr.y (mgr.14144) 152 : cephadm [INF] Saving service prometheus spec with placement smithi179=a;count:1 2024-09-17T07:12:19.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:18.077851+0000 mon.a (mon.0) 453 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:19.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:18.078684+0000 mon.a (mon.0) 454 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:12:19.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:18.295415+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:19.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:12:18.306687+0000 mgr.y (mgr.14144) 153 : cephadm [INF] Deploying daemon prometheus.a on smithi179 2024-09-17T07:12:19.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:17.765799+0000 mgr.y (mgr.14144) 150 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:19.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:18.072431+0000 mgr.y (mgr.14144) 151 : audit [DBG] from='client.34313 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi179=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:12:19.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:12:18.073193+0000 mgr.y (mgr.14144) 152 : cephadm [INF] Saving service prometheus spec with placement smithi179=a;count:1 2024-09-17T07:12:19.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:18.077851+0000 mon.a (mon.0) 453 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:19.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:18.078684+0000 mon.a (mon.0) 454 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:12:19.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:18.295415+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:19.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:12:18.306687+0000 mgr.y (mgr.14144) 153 : cephadm [INF] Deploying daemon prometheus.a on smithi179 2024-09-17T07:12:20.609 INFO:teuthology.orchestra.run.smithi179.stdout:Scheduled node-exporter update... 2024-09-17T07:12:20.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:19.766129+0000 mgr.y (mgr.14144) 154 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:20.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:20.608550+0000 mon.a (mon.0) 456 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:21.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:19.766129+0000 mgr.y (mgr.14144) 154 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:21.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:20.608550+0000 mon.a (mon.0) 456 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:21.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:19.766129+0000 mgr.y (mgr.14144) 154 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:21.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:20.608550+0000 mon.a (mon.0) 456 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:21.239 DEBUG:teuthology.orchestra.run.smithi044:node-exporter.a> sudo journalctl -f -n 0 -u ceph-817ae996-74c3-11ef-bceb-c7b262605968@node-exporter.a.service 2024-09-17T07:12:21.242 DEBUG:teuthology.orchestra.run.smithi179:node-exporter.b> sudo journalctl -f -n 0 -u ceph-817ae996-74c3-11ef-bceb-c7b262605968@node-exporter.b.service 2024-09-17T07:12:21.245 INFO:tasks.cephadm:Adding alertmanager.a on smithi044 2024-09-17T07:12:21.245 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph orch apply alertmanager '1;smithi044=a' 2024-09-17T07:12:21.827 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:21 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:20.603792+0000 mgr.y (mgr.14144) 155 : audit [DBG] from='client.34319 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi044=a;smithi179=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:12:21.827 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:21 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:12:20.604605+0000 mgr.y (mgr.14144) 156 : cephadm [INF] Saving service node-exporter spec with placement smithi044=a;smithi179=b;count:2 2024-09-17T07:12:22.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:21 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:20.603792+0000 mgr.y (mgr.14144) 155 : audit [DBG] from='client.34319 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi044=a;smithi179=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:12:22.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:21 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:12:20.604605+0000 mgr.y (mgr.14144) 156 : cephadm [INF] Saving service node-exporter spec with placement smithi044=a;smithi179=b;count:2 2024-09-17T07:12:22.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:21 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:20.603792+0000 mgr.y (mgr.14144) 155 : audit [DBG] from='client.34319 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi044=a;smithi179=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:12:22.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:21 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:12:20.604605+0000 mgr.y (mgr.14144) 156 : cephadm [INF] Saving service node-exporter spec with placement smithi044=a;smithi179=b;count:2 2024-09-17T07:12:22.783 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:21.766464+0000 mgr.y (mgr.14144) 157 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:23.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:22 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:21.766464+0000 mgr.y (mgr.14144) 157 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:23.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:22 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:21.766464+0000 mgr.y (mgr.14144) 157 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:23.197 INFO:teuthology.orchestra.run.smithi179.stdout:Scheduled alertmanager update... 2024-09-17T07:12:23.798 DEBUG:teuthology.orchestra.run.smithi044:alertmanager.a> sudo journalctl -f -n 0 -u ceph-817ae996-74c3-11ef-bceb-c7b262605968@alertmanager.a.service 2024-09-17T07:12:23.802 INFO:tasks.cephadm:Adding grafana.a on smithi179 2024-09-17T07:12:23.802 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph orch apply grafana '1;smithi179=a' 2024-09-17T07:12:24.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:24 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:23.190857+0000 mgr.y (mgr.14144) 158 : audit [DBG] from='client.34325 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi044=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:12:24.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:24 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:12:23.191601+0000 mgr.y (mgr.14144) 159 : cephadm [INF] Saving service alertmanager spec with placement smithi044=a;count:1 2024-09-17T07:12:24.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:24 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:23.196401+0000 mon.a (mon.0) 457 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:24.574 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:23.190857+0000 mgr.y (mgr.14144) 158 : audit [DBG] from='client.34325 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi044=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:12:24.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:12:23.191601+0000 mgr.y (mgr.14144) 159 : cephadm [INF] Saving service alertmanager spec with placement smithi044=a;count:1 2024-09-17T07:12:24.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:23.196401+0000 mon.a (mon.0) 457 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:24.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:23.190857+0000 mgr.y (mgr.14144) 158 : audit [DBG] from='client.34325 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi044=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:12:24.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:12:23.191601+0000 mgr.y (mgr.14144) 159 : cephadm [INF] Saving service alertmanager spec with placement smithi044=a;count:1 2024-09-17T07:12:24.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:23.196401+0000 mon.a (mon.0) 457 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:25.300 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:25 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:23.766938+0000 mgr.y (mgr.14144) 160 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:25.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:23.766938+0000 mgr.y (mgr.14144) 160 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:25.574 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:23.766938+0000 mgr.y (mgr.14144) 160 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:25.765 INFO:teuthology.orchestra.run.smithi179.stdout:Scheduled grafana update... 2024-09-17T07:12:26.809 DEBUG:teuthology.orchestra.run.smithi179:grafana.a> sudo journalctl -f -n 0 -u ceph-817ae996-74c3-11ef-bceb-c7b262605968@grafana.a.service 2024-09-17T07:12:26.811 INFO:tasks.cephadm:Setting up client nodes... 2024-09-17T07:12:26.811 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-17T07:12:27.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:25.759799+0000 mgr.y (mgr.14144) 161 : audit [DBG] from='client.34331 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi179=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:12:27.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:12:25.760541+0000 mgr.y (mgr.14144) 162 : cephadm [INF] Saving service grafana spec with placement smithi179=a;count:1 2024-09-17T07:12:27.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:25.764442+0000 mon.a (mon.0) 458 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:27.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:25.767224+0000 mgr.y (mgr.14144) 163 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:27.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:25.759799+0000 mgr.y (mgr.14144) 161 : audit [DBG] from='client.34331 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi179=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:12:27.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:12:25.760541+0000 mgr.y (mgr.14144) 162 : cephadm [INF] Saving service grafana spec with placement smithi179=a;count:1 2024-09-17T07:12:27.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:25.764442+0000 mon.a (mon.0) 458 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:27.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:25.767224+0000 mgr.y (mgr.14144) 163 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:27.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:26 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:25.759799+0000 mgr.y (mgr.14144) 161 : audit [DBG] from='client.34331 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi179=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:12:27.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:26 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:12:25.760541+0000 mgr.y (mgr.14144) 162 : cephadm [INF] Saving service grafana spec with placement smithi179=a;count:1 2024-09-17T07:12:27.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:26 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:25.764442+0000 mon.a (mon.0) 458 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:27.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:26 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:25.767224+0000 mgr.y (mgr.14144) 163 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:28.812 INFO:teuthology.orchestra.run.smithi044.stdout:[client.0] 2024-09-17T07:12:28.812 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQDcK+lm5N8uMBAAT/jar0S/lFbbKcRlOLjsnQ== 2024-09-17T07:12:29.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:28 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:27.767680+0000 mgr.y (mgr.14144) 164 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:29.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:28 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:28.807930+0000 mon.c (mon.1) 10 : audit [INF] from='client.? 172.21.15.44:0/1087254258' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T07:12:29.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:28 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:28.808227+0000 mon.a (mon.0) 459 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T07:12:29.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:28 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:28.811363+0000 mon.a (mon.0) 460 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T07:12:29.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:28 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:27.767680+0000 mgr.y (mgr.14144) 164 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:29.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:28 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:28.807930+0000 mon.c (mon.1) 10 : audit [INF] from='client.? 172.21.15.44:0/1087254258' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T07:12:29.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:28 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:28.808227+0000 mon.a (mon.0) 459 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T07:12:29.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:28 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:28.811363+0000 mon.a (mon.0) 460 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T07:12:29.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:28 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:27.767680+0000 mgr.y (mgr.14144) 164 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:29.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:28 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:28.807930+0000 mon.c (mon.1) 10 : audit [INF] from='client.? 172.21.15.44:0/1087254258' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T07:12:29.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:28 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:28.808227+0000 mon.a (mon.0) 459 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T07:12:29.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:28 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:28.811363+0000 mon.a (mon.0) 460 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T07:12:29.487 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-09-17T07:12:29.487 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-17T07:12:29.487 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-17T07:12:29.530 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-17T07:12:30.950 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:30 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:29.768217+0000 mgr.y (mgr.14144) 165 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:31.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:29.768217+0000 mgr.y (mgr.14144) 165 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:31.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:29.768217+0000 mgr.y (mgr.14144) 165 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:33.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:32 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:31.768603+0000 mgr.y (mgr.14144) 166 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:33.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:32 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:31.768603+0000 mgr.y (mgr.14144) 166 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:33.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:31.768603+0000 mgr.y (mgr.14144) 166 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:34.987 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:33.769070+0000 mgr.y (mgr.14144) 167 : cluster [DBG] pgmap v153: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:35.207 INFO:teuthology.orchestra.run.smithi179.stdout:[client.1] 2024-09-17T07:12:35.207 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQDjK+lmh5kODBAATOZxtjhJmIWOIMy7q3FZWw== 2024-09-17T07:12:35.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:33.769070+0000 mgr.y (mgr.14144) 167 : cluster [DBG] pgmap v153: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:35.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:33.769070+0000 mgr.y (mgr.14144) 167 : cluster [DBG] pgmap v153: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:35.922 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:35 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:35.201456+0000 mon.b (mon.2) 22 : audit [INF] from='client.? 172.21.15.179:0/1006174752' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T07:12:35.922 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:35 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:35.202118+0000 mon.a (mon.0) 461 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T07:12:35.922 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:35 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:35.205656+0000 mon.a (mon.0) 462 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T07:12:35.923 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-09-17T07:12:35.923 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-17T07:12:35.924 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-17T07:12:35.965 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-17T07:12:35.965 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-17T07:12:35.966 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-17T07:12:36.182 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/817ae996-74c3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:12:36.187 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:12:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[35786]: level=info ts=2024-09-17T07:12:36.109Z caller=main.go:646 msg="Server is ready to receive web requests." 2024-09-17T07:12:36.187 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:12:36 smithi179 bash[35756]: 556d0546f78eea557b88ac45406fb8e874a1a446f267284df556444c036efdd3 2024-09-17T07:12:36.304 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:35.201456+0000 mon.b (mon.2) 22 : audit [INF] from='client.? 172.21.15.179:0/1006174752' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T07:12:36.304 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:35.202118+0000 mon.a (mon.0) 461 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T07:12:36.304 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:35.205656+0000 mon.a (mon.0) 462 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T07:12:36.305 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:35.201456+0000 mon.b (mon.2) 22 : audit [INF] from='client.? 172.21.15.179:0/1006174752' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T07:12:36.305 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:35.202118+0000 mon.a (mon.0) 461 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T07:12:36.305 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:35.205656+0000 mon.a (mon.0) 462 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T07:12:36.614 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:12:36 smithi179 systemd[1]: Started Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:12:37.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:37 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:35.769447+0000 mgr.y (mgr.14144) 168 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:37.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:37 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:36.247393+0000 mon.a (mon.0) 463 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:37.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:37 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:36.248034+0000 mon.a (mon.0) 464 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:12:37.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:35.769447+0000 mgr.y (mgr.14144) 168 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:37.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:36.247393+0000 mon.a (mon.0) 463 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:37.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:36.248034+0000 mon.a (mon.0) 464 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:12:37.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:35.769447+0000 mgr.y (mgr.14144) 168 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:37.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:36.247393+0000 mon.a (mon.0) 463 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:37.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:36.248034+0000 mon.a (mon.0) 464 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:12:38.061 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-09-17T07:12:38.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:38.056766+0000 mon.c (mon.1) 11 : audit [DBG] from='client.? 172.21.15.44:0/2814080598' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-17T07:12:38.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:38.076792+0000 mon.a (mon.0) 465 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:38.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:38.081854+0000 mon.a (mon.0) 466 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:38.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:38.056766+0000 mon.c (mon.1) 11 : audit [DBG] from='client.? 172.21.15.44:0/2814080598' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-17T07:12:38.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:38.076792+0000 mon.a (mon.0) 465 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:38.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:38.081854+0000 mon.a (mon.0) 466 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:38.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:38 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:38.056766+0000 mon.c (mon.1) 11 : audit [DBG] from='client.? 172.21.15.44:0/2814080598' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-17T07:12:38.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:38 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:38.076792+0000 mon.a (mon.0) 465 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:38.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:38 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:38.081854+0000 mon.a (mon.0) 466 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:38.690 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":14,"active_gid":14144,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":3175338091},{"type":"v1","addr":"172.21.15.44:6801","nonce":3175338091}]},"active_addr":"172.21.15.44:6801/3175338091","active_change":"2024-09-17T07:08:47.676869+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[{"gid":34109,"name":"x","mgr_features":4540138297136906239,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://smithi044.front.sepia.ceph.com:8443/"},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":3,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.44:0","nonce":3964501701}]},{"addrvec":[{"type":"v2","addr":"172.21.15.44:0","nonce":512688273}]}]}} 2024-09-17T07:12:38.694 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-17T07:12:38.694 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-17T07:12:38.694 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-17T07:12:38.909 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/817ae996-74c3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:12:39.114 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:38 smithi179 bash[36215]: Trying to pull docker.io/prom/node-exporter:v0.18.1... 2024-09-17T07:12:39.289 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:37.769728+0000 mgr.y (mgr.14144) 169 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:39.290 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:12:38.083319+0000 mgr.y (mgr.14144) 170 : cephadm [INF] Deploying daemon node-exporter.b on smithi179 2024-09-17T07:12:39.290 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:37.769728+0000 mgr.y (mgr.14144) 169 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:39.290 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:12:38.083319+0000 mgr.y (mgr.14144) 170 : cephadm [INF] Deploying daemon node-exporter.b on smithi179 2024-09-17T07:12:39.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:37.769728+0000 mgr.y (mgr.14144) 169 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:39.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:12:38.083319+0000 mgr.y (mgr.14144) 170 : cephadm [INF] Deploying daemon node-exporter.b on smithi179 2024-09-17T07:12:40.676 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-09-17T07:12:40.676 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":50,"fsid":"817ae996-74c3-11ef-bceb-c7b262605968","created":"2024-09-17T07:08:04.408168+0000","modified":"2024-09-17T07:12:10.632496+0000","last_up_change":"2024-09-17T07:12:09.510313+0000","last_in_change":"2024-09-17T07:11:54.001817+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-09-17T07:10:35.776015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"7f236893-2f40-4119-ab88-7359969cc30a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":1979937622},{"type":"v1","addr":"172.21.15.44:6803","nonce":1979937622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":1979937622},{"type":"v1","addr":"172.21.15.44:6805","nonce":1979937622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":1979937622},{"type":"v1","addr":"172.21.15.44:6809","nonce":1979937622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":1979937622},{"type":"v1","addr":"172.21.15.44:6807","nonce":1979937622}]},"public_addr":"172.21.15.44:6803/1979937622","cluster_addr":"172.21.15.44:6805/1979937622","heartbeat_back_addr":"172.21.15.44:6809/1979937622","heartbeat_front_addr":"172.21.15.44:6807/1979937622","state":["exists","up"]},{"osd":1,"uuid":"f8da14c0-ab31-42e2-be11-bd734d990923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":1594621847},{"type":"v1","addr":"172.21.15.44:6811","nonce":1594621847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":1594621847},{"type":"v1","addr":"172.21.15.44:6813","nonce":1594621847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":1594621847},{"type":"v1","addr":"172.21.15.44:6817","nonce":1594621847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":1594621847},{"type":"v1","addr":"172.21.15.44:6815","nonce":1594621847}]},"public_addr":"172.21.15.44:6811/1594621847","cluster_addr":"172.21.15.44:6813/1594621847","heartbeat_back_addr":"172.21.15.44:6817/1594621847","heartbeat_front_addr":"172.21.15.44:6815/1594621847","state":["exists","up"]},{"osd":2,"uuid":"68dff20f-443b-4ed3-86a5-dc84cb5b9da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":3871662332},{"type":"v1","addr":"172.21.15.44:6819","nonce":3871662332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":3871662332},{"type":"v1","addr":"172.21.15.44:6821","nonce":3871662332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":3871662332},{"type":"v1","addr":"172.21.15.44:6825","nonce":3871662332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":3871662332},{"type":"v1","addr":"172.21.15.44:6823","nonce":3871662332}]},"public_addr":"172.21.15.44:6819/3871662332","cluster_addr":"172.21.15.44:6821/3871662332","heartbeat_back_addr":"172.21.15.44:6825/3871662332","heartbeat_front_addr":"172.21.15.44:6823/3871662332","state":["exists","up"]},{"osd":3,"uuid":"72f5c16b-cdf4-45fa-a49b-93fed3cf8d07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":905450488},{"type":"v1","addr":"172.21.15.44:6827","nonce":905450488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":905450488},{"type":"v1","addr":"172.21.15.44:6829","nonce":905450488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":905450488},{"type":"v1","addr":"172.21.15.44:6833","nonce":905450488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":905450488},{"type":"v1","addr":"172.21.15.44:6831","nonce":905450488}]},"public_addr":"172.21.15.44:6827/905450488","cluster_addr":"172.21.15.44:6829/905450488","heartbeat_back_addr":"172.21.15.44:6833/905450488","heartbeat_front_addr":"172.21.15.44:6831/905450488","state":["exists","up"]},{"osd":4,"uuid":"ff638478-3d19-4b13-9123-4be3ad21a531","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2112811328},{"type":"v1","addr":"172.21.15.179:6801","nonce":2112811328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2112811328},{"type":"v1","addr":"172.21.15.179:6803","nonce":2112811328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2112811328},{"type":"v1","addr":"172.21.15.179:6807","nonce":2112811328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2112811328},{"type":"v1","addr":"172.21.15.179:6805","nonce":2112811328}]},"public_addr":"172.21.15.179:6801/2112811328","cluster_addr":"172.21.15.179:6803/2112811328","heartbeat_back_addr":"172.21.15.179:6807/2112811328","heartbeat_front_addr":"172.21.15.179:6805/2112811328","state":["exists","up"]},{"osd":5,"uuid":"83cd743c-9dcb-4027-8857-34773603b77c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":3243120327},{"type":"v1","addr":"172.21.15.179:6809","nonce":3243120327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":3243120327},{"type":"v1","addr":"172.21.15.179:6811","nonce":3243120327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":3243120327},{"type":"v1","addr":"172.21.15.179:6815","nonce":3243120327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":3243120327},{"type":"v1","addr":"172.21.15.179:6813","nonce":3243120327}]},"public_addr":"172.21.15.179:6809/3243120327","cluster_addr":"172.21.15.179:6811/3243120327","heartbeat_back_addr":"172.21.15.179:6815/3243120327","heartbeat_front_addr":"172.21.15.179:6813/3243120327","state":["exists","up"]},{"osd":6,"uuid":"4e608f05-0fef-407f-aa40-ded55acd0bb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":1852137393},{"type":"v1","addr":"172.21.15.179:6817","nonce":1852137393}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":1852137393},{"type":"v1","addr":"172.21.15.179:6819","nonce":1852137393}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":1852137393},{"type":"v1","addr":"172.21.15.179:6823","nonce":1852137393}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":1852137393},{"type":"v1","addr":"172.21.15.179:6821","nonce":1852137393}]},"public_addr":"172.21.15.179:6817/1852137393","cluster_addr":"172.21.15.179:6819/1852137393","heartbeat_back_addr":"172.21.15.179:6823/1852137393","heartbeat_front_addr":"172.21.15.179:6821/1852137393","state":["exists","up"]},{"osd":7,"uuid":"d1cfecfc-fb47-4667-939a-adb55b684468","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":191862342},{"type":"v1","addr":"172.21.15.179:6825","nonce":191862342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":191862342},{"type":"v1","addr":"172.21.15.179:6827","nonce":191862342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":191862342},{"type":"v1","addr":"172.21.15.179:6831","nonce":191862342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":191862342},{"type":"v1","addr":"172.21.15.179:6829","nonce":191862342}]},"public_addr":"172.21.15.179:6825/191862342","cluster_addr":"172.21.15.179:6827/191862342","heartbeat_back_addr":"172.21.15.179:6831/191862342","heartbeat_front_addr":"172.21.15.179:6829/191862342","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:09:55.262575+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:10:13.977151+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:10:33.368706+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:10:53.567070+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:11:09.924018+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:11:29.294199+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:11:48.674903+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:12:07.566776+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/3961128045":"2024-09-18T07:08:47.676584+0000","172.21.15.44:0/367718499":"2024-09-18T07:08:47.676584+0000","172.21.15.44:6800/930976792":"2024-09-18T07:08:47.676584+0000","172.21.15.44:6801/930976792":"2024-09-18T07:08:47.676584+0000","172.21.15.44:0/1289983812":"2024-09-18T07:08:26.744189+0000","172.21.15.44:0/300796373":"2024-09-18T07:08:26.744189+0000","172.21.15.44:6801/2448217567":"2024-09-18T07:08:26.744189+0000","172.21.15.44:6800/2448217567":"2024-09-18T07:08:26.744189+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T07:12:40.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:39.770189+0000 mgr.y (mgr.14144) 171 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:40.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:39.770189+0000 mgr.y (mgr.14144) 171 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:41.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:40 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:39.770189+0000 mgr.y (mgr.14144) 171 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:41.407 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-17T07:12:41.408 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-17T07:12:41.621 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/817ae996-74c3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:12:41.967 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:41 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:40.675382+0000 mon.c (mon.1) 12 : audit [DBG] from='client.? 172.21.15.44:0/3375968121' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T07:12:41.967 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:41 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:40.675382+0000 mon.c (mon.1) 12 : audit [DBG] from='client.? 172.21.15.44:0/3375968121' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T07:12:42.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:41 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:40.675382+0000 mon.c (mon.1) 12 : audit [DBG] from='client.? 172.21.15.44:0/3375968121' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T07:12:43.004 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:41.770563+0000 mgr.y (mgr.14144) 172 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:43.004 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:41.770563+0000 mgr.y (mgr.14144) 172 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:43.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:41.770563+0000 mgr.y (mgr.14144) 172 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:43.400 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-09-17T07:12:43.400 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":50,"fsid":"817ae996-74c3-11ef-bceb-c7b262605968","created":"2024-09-17T07:08:04.408168+0000","modified":"2024-09-17T07:12:10.632496+0000","last_up_change":"2024-09-17T07:12:09.510313+0000","last_in_change":"2024-09-17T07:11:54.001817+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-09-17T07:10:35.776015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"7f236893-2f40-4119-ab88-7359969cc30a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":1979937622},{"type":"v1","addr":"172.21.15.44:6803","nonce":1979937622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":1979937622},{"type":"v1","addr":"172.21.15.44:6805","nonce":1979937622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":1979937622},{"type":"v1","addr":"172.21.15.44:6809","nonce":1979937622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":1979937622},{"type":"v1","addr":"172.21.15.44:6807","nonce":1979937622}]},"public_addr":"172.21.15.44:6803/1979937622","cluster_addr":"172.21.15.44:6805/1979937622","heartbeat_back_addr":"172.21.15.44:6809/1979937622","heartbeat_front_addr":"172.21.15.44:6807/1979937622","state":["exists","up"]},{"osd":1,"uuid":"f8da14c0-ab31-42e2-be11-bd734d990923","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":1594621847},{"type":"v1","addr":"172.21.15.44:6811","nonce":1594621847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":1594621847},{"type":"v1","addr":"172.21.15.44:6813","nonce":1594621847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":1594621847},{"type":"v1","addr":"172.21.15.44:6817","nonce":1594621847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":1594621847},{"type":"v1","addr":"172.21.15.44:6815","nonce":1594621847}]},"public_addr":"172.21.15.44:6811/1594621847","cluster_addr":"172.21.15.44:6813/1594621847","heartbeat_back_addr":"172.21.15.44:6817/1594621847","heartbeat_front_addr":"172.21.15.44:6815/1594621847","state":["exists","up"]},{"osd":2,"uuid":"68dff20f-443b-4ed3-86a5-dc84cb5b9da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":3871662332},{"type":"v1","addr":"172.21.15.44:6819","nonce":3871662332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":3871662332},{"type":"v1","addr":"172.21.15.44:6821","nonce":3871662332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":3871662332},{"type":"v1","addr":"172.21.15.44:6825","nonce":3871662332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":3871662332},{"type":"v1","addr":"172.21.15.44:6823","nonce":3871662332}]},"public_addr":"172.21.15.44:6819/3871662332","cluster_addr":"172.21.15.44:6821/3871662332","heartbeat_back_addr":"172.21.15.44:6825/3871662332","heartbeat_front_addr":"172.21.15.44:6823/3871662332","state":["exists","up"]},{"osd":3,"uuid":"72f5c16b-cdf4-45fa-a49b-93fed3cf8d07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":905450488},{"type":"v1","addr":"172.21.15.44:6827","nonce":905450488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":905450488},{"type":"v1","addr":"172.21.15.44:6829","nonce":905450488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":905450488},{"type":"v1","addr":"172.21.15.44:6833","nonce":905450488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":905450488},{"type":"v1","addr":"172.21.15.44:6831","nonce":905450488}]},"public_addr":"172.21.15.44:6827/905450488","cluster_addr":"172.21.15.44:6829/905450488","heartbeat_back_addr":"172.21.15.44:6833/905450488","heartbeat_front_addr":"172.21.15.44:6831/905450488","state":["exists","up"]},{"osd":4,"uuid":"ff638478-3d19-4b13-9123-4be3ad21a531","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2112811328},{"type":"v1","addr":"172.21.15.179:6801","nonce":2112811328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2112811328},{"type":"v1","addr":"172.21.15.179:6803","nonce":2112811328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2112811328},{"type":"v1","addr":"172.21.15.179:6807","nonce":2112811328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2112811328},{"type":"v1","addr":"172.21.15.179:6805","nonce":2112811328}]},"public_addr":"172.21.15.179:6801/2112811328","cluster_addr":"172.21.15.179:6803/2112811328","heartbeat_back_addr":"172.21.15.179:6807/2112811328","heartbeat_front_addr":"172.21.15.179:6805/2112811328","state":["exists","up"]},{"osd":5,"uuid":"83cd743c-9dcb-4027-8857-34773603b77c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":3243120327},{"type":"v1","addr":"172.21.15.179:6809","nonce":3243120327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":3243120327},{"type":"v1","addr":"172.21.15.179:6811","nonce":3243120327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":3243120327},{"type":"v1","addr":"172.21.15.179:6815","nonce":3243120327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":3243120327},{"type":"v1","addr":"172.21.15.179:6813","nonce":3243120327}]},"public_addr":"172.21.15.179:6809/3243120327","cluster_addr":"172.21.15.179:6811/3243120327","heartbeat_back_addr":"172.21.15.179:6815/3243120327","heartbeat_front_addr":"172.21.15.179:6813/3243120327","state":["exists","up"]},{"osd":6,"uuid":"4e608f05-0fef-407f-aa40-ded55acd0bb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":1852137393},{"type":"v1","addr":"172.21.15.179:6817","nonce":1852137393}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":1852137393},{"type":"v1","addr":"172.21.15.179:6819","nonce":1852137393}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":1852137393},{"type":"v1","addr":"172.21.15.179:6823","nonce":1852137393}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":1852137393},{"type":"v1","addr":"172.21.15.179:6821","nonce":1852137393}]},"public_addr":"172.21.15.179:6817/1852137393","cluster_addr":"172.21.15.179:6819/1852137393","heartbeat_back_addr":"172.21.15.179:6823/1852137393","heartbeat_front_addr":"172.21.15.179:6821/1852137393","state":["exists","up"]},{"osd":7,"uuid":"d1cfecfc-fb47-4667-939a-adb55b684468","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":191862342},{"type":"v1","addr":"172.21.15.179:6825","nonce":191862342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":191862342},{"type":"v1","addr":"172.21.15.179:6827","nonce":191862342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":191862342},{"type":"v1","addr":"172.21.15.179:6831","nonce":191862342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":191862342},{"type":"v1","addr":"172.21.15.179:6829","nonce":191862342}]},"public_addr":"172.21.15.179:6825/191862342","cluster_addr":"172.21.15.179:6827/191862342","heartbeat_back_addr":"172.21.15.179:6831/191862342","heartbeat_front_addr":"172.21.15.179:6829/191862342","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:09:55.262575+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:10:13.977151+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:10:33.368706+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:10:53.567070+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:11:09.924018+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:11:29.294199+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:11:48.674903+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T07:12:07.566776+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/3961128045":"2024-09-18T07:08:47.676584+0000","172.21.15.44:0/367718499":"2024-09-18T07:08:47.676584+0000","172.21.15.44:6800/930976792":"2024-09-18T07:08:47.676584+0000","172.21.15.44:6801/930976792":"2024-09-18T07:08:47.676584+0000","172.21.15.44:0/1289983812":"2024-09-18T07:08:26.744189+0000","172.21.15.44:0/300796373":"2024-09-18T07:08:26.744189+0000","172.21.15.44:6801/2448217567":"2024-09-18T07:08:26.744189+0000","172.21.15.44:6800/2448217567":"2024-09-18T07:08:26.744189+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T07:12:43.742 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:43 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:43.399727+0000 mon.c (mon.1) 13 : audit [DBG] from='client.? 172.21.15.44:0/146571990' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T07:12:43.743 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:43 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:43.399727+0000 mon.c (mon.1) 13 : audit [DBG] from='client.? 172.21.15.44:0/146571990' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T07:12:44.081 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-17T07:12:44.081 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-17T07:12:44.082 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-17T07:12:44.082 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-17T07:12:44.082 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-17T07:12:44.083 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-17T07:12:44.083 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-17T07:12:44.084 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-17T07:12:44.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:43 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:43.399727+0000 mon.c (mon.1) 13 : audit [DBG] from='client.? 172.21.15.44:0/146571990' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T07:12:44.491 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/817ae996-74c3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:12:44.491 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/817ae996-74c3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:12:44.491 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/817ae996-74c3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:12:44.493 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/817ae996-74c3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:12:44.493 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/817ae996-74c3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:12:44.493 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/817ae996-74c3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:12:44.494 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/817ae996-74c3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:12:44.497 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/817ae996-74c3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:12:44.934 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:43.770976+0000 mgr.y (mgr.14144) 173 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:44.934 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:43.770976+0000 mgr.y (mgr.14144) 173 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:45.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:43.770976+0000 mgr.y (mgr.14144) 173 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:45.614 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:45 smithi179 bash[36215]: Getting image source signatures 2024-09-17T07:12:45.614 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:45 smithi179 bash[36215]: Copying blob sha256:190160031744b0810999dd6f293478be0b41524654c70782cf09437095430f2f 2024-09-17T07:12:45.614 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:45 smithi179 bash[36215]: Copying blob sha256:49a2d53aa1af115e87f7e23a7fe4b7a3e12304d37982fb6528b035d0f68a3b56 2024-09-17T07:12:45.614 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:45 smithi179 bash[36215]: Copying blob sha256:3589a6efd9ce96b2f2218a80bacb84f60d5e584d024023617b0f56b87fe798c3 2024-09-17T07:12:47.109 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:45.771252+0000 mgr.y (mgr.14144) 174 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:47.110 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:45.771252+0000 mgr.y (mgr.14144) 174 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:47.113 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:45.771252+0000 mgr.y (mgr.14144) 174 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:47.114 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:46 smithi179 bash[36215]: Copying config sha256:e5a616e4b9cf68dfcad7782b78e118be4310022e874d52da85c55923fb615f87 2024-09-17T07:12:47.114 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:46 smithi179 bash[36215]: Writing manifest to image destination 2024-09-17T07:12:47.438 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 podman[36215]: 2024-09-17 07:12:47.10383249 +0000 UTC m=+8.402186779 image pull e5a616e4b9cf68dfcad7782b78e118be4310022e874d52da85c55923fb615f87 docker.io/prom/node-exporter:v0.18.1 2024-09-17T07:12:47.438 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 podman[36215]: 2024-09-17 07:12:47.327636706 +0000 UTC m=+8.625990981 container create 2106bcb6f0e8d97c5ee3169986fec7d4e8581f1e77ffd2fce466ea9e2c62ad56 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b, maintainer=The Prometheus Authors ) 2024-09-17T07:12:47.864 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 podman[36215]: 2024-09-17 07:12:47.546383199 +0000 UTC m=+8.844737488 container init 2106bcb6f0e8d97c5ee3169986fec7d4e8581f1e77ffd2fce466ea9e2c62ad56 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b, maintainer=The Prometheus Authors ) 2024-09-17T07:12:47.865 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 podman[36215]: 2024-09-17 07:12:47.550118067 +0000 UTC m=+8.848472350 container start 2106bcb6f0e8d97c5ee3169986fec7d4e8581f1e77ffd2fce466ea9e2c62ad56 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b, maintainer=The Prometheus Authors ) 2024-09-17T07:12:47.865 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[36291]: time="2024-09-17T07:12:47Z" level=info msg="Starting node_exporter (version=0.18.1, branch=HEAD, revision=3db77732e925c08f675d7404a8c46466b2ece83e)" source="node_exporter.go:156" 2024-09-17T07:12:47.865 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[36291]: time="2024-09-17T07:12:47Z" level=info msg="Build context (go=go1.12.5, user=root@b50852a1acba, date=20190604-16:41:18)" source="node_exporter.go:157" 2024-09-17T07:12:47.865 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[36291]: time="2024-09-17T07:12:47Z" level=info msg="Enabled collectors:" source="node_exporter.go:97" 2024-09-17T07:12:47.865 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[36291]: time="2024-09-17T07:12:47Z" level=info msg=" - arp" source="node_exporter.go:104" 2024-09-17T07:12:47.865 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[36291]: time="2024-09-17T07:12:47Z" level=info msg=" - bcache" source="node_exporter.go:104" 2024-09-17T07:12:47.865 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[36291]: time="2024-09-17T07:12:47Z" level=info msg=" - bonding" source="node_exporter.go:104" 2024-09-17T07:12:47.865 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[36291]: time="2024-09-17T07:12:47Z" level=info msg=" - conntrack" source="node_exporter.go:104" 2024-09-17T07:12:47.865 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[36291]: time="2024-09-17T07:12:47Z" level=info msg=" - cpu" source="node_exporter.go:104" 2024-09-17T07:12:47.865 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[36291]: time="2024-09-17T07:12:47Z" level=info msg=" - cpufreq" source="node_exporter.go:104" 2024-09-17T07:12:47.866 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[36291]: time="2024-09-17T07:12:47Z" level=info msg=" - diskstats" source="node_exporter.go:104" 2024-09-17T07:12:47.866 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[36291]: time="2024-09-17T07:12:47Z" level=info msg=" - edac" source="node_exporter.go:104" 2024-09-17T07:12:47.866 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[36291]: time="2024-09-17T07:12:47Z" level=info msg=" - entropy" source="node_exporter.go:104" 2024-09-17T07:12:47.866 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[36291]: time="2024-09-17T07:12:47Z" level=info msg=" - filefd" source="node_exporter.go:104" 2024-09-17T07:12:47.866 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[36291]: time="2024-09-17T07:12:47Z" level=info msg=" - filesystem" source="node_exporter.go:104" 2024-09-17T07:12:47.866 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[36291]: time="2024-09-17T07:12:47Z" level=info msg=" - hwmon" source="node_exporter.go:104" 2024-09-17T07:12:47.866 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[36291]: time="2024-09-17T07:12:47Z" level=info msg=" - infiniband" source="node_exporter.go:104" 2024-09-17T07:12:47.866 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[36291]: time="2024-09-17T07:12:47Z" level=info msg=" - ipvs" source="node_exporter.go:104" 2024-09-17T07:12:47.866 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[36291]: time="2024-09-17T07:12:47Z" level=info msg=" - loadavg" source="node_exporter.go:104" 2024-09-17T07:12:47.866 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[36291]: time="2024-09-17T07:12:47Z" level=info msg=" - mdadm" source="node_exporter.go:104" 2024-09-17T07:12:47.866 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[36291]: time="2024-09-17T07:12:47Z" level=info msg=" - meminfo" source="node_exporter.go:104" 2024-09-17T07:12:47.866 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[36291]: time="2024-09-17T07:12:47Z" level=info msg=" - netclass" source="node_exporter.go:104" 2024-09-17T07:12:47.866 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[36291]: time="2024-09-17T07:12:47Z" level=info msg=" - netdev" source="node_exporter.go:104" 2024-09-17T07:12:47.866 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[36291]: time="2024-09-17T07:12:47Z" level=info msg=" - netstat" source="node_exporter.go:104" 2024-09-17T07:12:47.866 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[36291]: time="2024-09-17T07:12:47Z" level=info msg=" - nfs" source="node_exporter.go:104" 2024-09-17T07:12:47.867 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[36291]: time="2024-09-17T07:12:47Z" level=info msg=" - nfsd" source="node_exporter.go:104" 2024-09-17T07:12:47.867 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[36291]: time="2024-09-17T07:12:47Z" level=info msg=" - pressure" source="node_exporter.go:104" 2024-09-17T07:12:47.867 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[36291]: time="2024-09-17T07:12:47Z" level=info msg=" - sockstat" source="node_exporter.go:104" 2024-09-17T07:12:47.867 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[36291]: time="2024-09-17T07:12:47Z" level=info msg=" - stat" source="node_exporter.go:104" 2024-09-17T07:12:47.867 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[36291]: time="2024-09-17T07:12:47Z" level=info msg=" - textfile" source="node_exporter.go:104" 2024-09-17T07:12:47.867 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[36291]: time="2024-09-17T07:12:47Z" level=info msg=" - time" source="node_exporter.go:104" 2024-09-17T07:12:47.867 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[36291]: time="2024-09-17T07:12:47Z" level=info msg=" - uname" source="node_exporter.go:104" 2024-09-17T07:12:47.867 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[36291]: time="2024-09-17T07:12:47Z" level=info msg=" - vmstat" source="node_exporter.go:104" 2024-09-17T07:12:47.867 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[36291]: time="2024-09-17T07:12:47Z" level=info msg=" - xfs" source="node_exporter.go:104" 2024-09-17T07:12:47.867 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[36291]: time="2024-09-17T07:12:47Z" level=info msg=" - zfs" source="node_exporter.go:104" 2024-09-17T07:12:47.867 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[36291]: time="2024-09-17T07:12:47Z" level=info msg="Listening on :9100" source="node_exporter.go:170" 2024-09-17T07:12:47.867 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 bash[36215]: 2106bcb6f0e8d97c5ee3169986fec7d4e8581f1e77ffd2fce466ea9e2c62ad56 2024-09-17T07:12:47.867 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:12:47 smithi179 systemd[1]: Started Ceph node-exporter.b for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:12:49.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:47.697553+0000 mon.a (mon.0) 467 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:49.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:12:47.698384+0000 mgr.y (mgr.14144) 175 : cephadm [INF] Deploying daemon node-exporter.a on smithi044 2024-09-17T07:12:49.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:47.771535+0000 mgr.y (mgr.14144) 176 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:49.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:47.867255+0000 mon.a (mon.0) 468 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:12:49.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:12:47.868490+0000 mon.a (mon.0) 469 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:12:49.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:47.697553+0000 mon.a (mon.0) 467 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:49.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:12:47.698384+0000 mgr.y (mgr.14144) 175 : cephadm [INF] Deploying daemon node-exporter.a on smithi044 2024-09-17T07:12:49.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:47.771535+0000 mgr.y (mgr.14144) 176 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:49.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:47.867255+0000 mon.a (mon.0) 468 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:12:49.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:12:47.868490+0000 mon.a (mon.0) 469 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:12:49.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:47.697553+0000 mon.a (mon.0) 467 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:12:49.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:12:47.698384+0000 mgr.y (mgr.14144) 175 : cephadm [INF] Deploying daemon node-exporter.a on smithi044 2024-09-17T07:12:49.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:47.771535+0000 mgr.y (mgr.14144) 176 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:49.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:47.867255+0000 mon.a (mon.0) 468 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:12:49.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:12:47.868490+0000 mon.a (mon.0) 469 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:12:50.750 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:49.771829+0000 mgr.y (mgr.14144) 177 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:50.750 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:49.771829+0000 mgr.y (mgr.14144) 177 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:51.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:49.771829+0000 mgr.y (mgr.14144) 177 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:53.026 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:51.772135+0000 mgr.y (mgr.14144) 178 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:53.027 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:51.772135+0000 mgr.y (mgr.14144) 178 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:53.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:51.772135+0000 mgr.y (mgr.14144) 178 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:54.944 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:53.772421+0000 mgr.y (mgr.14144) 179 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:54.945 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:53.772421+0000 mgr.y (mgr.14144) 179 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:55.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:53.772421+0000 mgr.y (mgr.14144) 179 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:55.642 INFO:teuthology.orchestra.run.smithi044.stdout:206158430220 2024-09-17T07:12:55.643 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-17T07:12:57.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:56 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:55.772781+0000 mgr.y (mgr.14144) 180 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:57.306 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:55.772781+0000 mgr.y (mgr.14144) 180 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:57.306 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:55.772781+0000 mgr.y (mgr.14144) 180 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:57.555 INFO:teuthology.orchestra.run.smithi044.stdout:128849018903 2024-09-17T07:12:57.555 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-17T07:12:58.192 INFO:teuthology.orchestra.run.smithi044.stdout:34359738407 2024-09-17T07:12:58.193 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-17T07:12:59.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:12:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:57.773286+0000 mgr.y (mgr.14144) 181 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:59.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:12:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:57.773286+0000 mgr.y (mgr.14144) 181 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:59.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:12:58 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:57.773286+0000 mgr.y (mgr.14144) 181 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:12:59.881 INFO:teuthology.orchestra.run.smithi044.stdout:180388626448 2024-09-17T07:12:59.882 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-17T07:13:00.027 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/817ae996-74c3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:13:00.176 INFO:teuthology.orchestra.run.smithi044.stdout:154618822675 2024-09-17T07:13:00.176 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-17T07:13:00.713 INFO:teuthology.orchestra.run.smithi044.stdout:55834574883 2024-09-17T07:13:00.713 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-17T07:13:00.916 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:00 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:12:59.773637+0000 mgr.y (mgr.14144) 182 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:01.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:12:59.773637+0000 mgr.y (mgr.14144) 182 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:01.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:12:59.773637+0000 mgr.y (mgr.14144) 182 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:01.101 INFO:teuthology.orchestra.run.smithi044.stdout:107374182427 2024-09-17T07:13:01.102 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-17T07:13:01.308 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/817ae996-74c3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:13:01.309 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/817ae996-74c3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:13:01.310 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/817ae996-74c3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:13:01.317 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/817ae996-74c3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:13:02.943 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/817ae996-74c3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:13:03.042 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:01.773976+0000 mgr.y (mgr.14144) 183 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:03.043 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:01.773976+0000 mgr.y (mgr.14144) 183 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:03.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:01.773976+0000 mgr.y (mgr.14144) 183 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:03.496 INFO:teuthology.orchestra.run.smithi044.stdout:77309411360 2024-09-17T07:13:03.497 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-17T07:13:04.955 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:04 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:03.774276+0000 mgr.y (mgr.14144) 184 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:04.955 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:04 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:03.774276+0000 mgr.y (mgr.14144) 184 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:05.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:04 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:03.774276+0000 mgr.y (mgr.14144) 184 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:07.041 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:05.774527+0000 mgr.y (mgr.14144) 185 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:07.041 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:05.774527+0000 mgr.y (mgr.14144) 185 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:07.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:05.774527+0000 mgr.y (mgr.14144) 185 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:07.807 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/817ae996-74c3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:13:07.807 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/817ae996-74c3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:13:08.324 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:08 smithi044 bash[40117]: Copying blob sha256:3589a6efd9ce96b2f2218a80bacb84f60d5e584d024023617b0f56b87fe798c3 2024-09-17T07:13:08.663 INFO:teuthology.orchestra.run.smithi044.stdout:34359738409 2024-09-17T07:13:09.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:07.775009+0000 mgr.y (mgr.14144) 186 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:09.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:08.662782+0000 mon.c (mon.1) 14 : audit [DBG] from='client.? 172.21.15.44:0/3902012750' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-17T07:13:09.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:07.775009+0000 mgr.y (mgr.14144) 186 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:09.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:08.662782+0000 mon.c (mon.1) 14 : audit [DBG] from='client.? 172.21.15.44:0/3902012750' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-17T07:13:09.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:07.775009+0000 mgr.y (mgr.14144) 186 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:09.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:08.662782+0000 mon.c (mon.1) 14 : audit [DBG] from='client.? 172.21.15.44:0/3902012750' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-17T07:13:09.619 INFO:teuthology.orchestra.run.smithi044.stdout:180388626450 2024-09-17T07:13:09.838 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:09 smithi044 bash[40117]: Copying config sha256:e5a616e4b9cf68dfcad7782b78e118be4310022e874d52da85c55923fb615f87 2024-09-17T07:13:10.102 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:10 smithi044 bash[40117]: Writing manifest to image destination 2024-09-17T07:13:10.102 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:09.576665+0000 mon.c (mon.1) 15 : audit [DBG] from='client.? 172.21.15.44:0/2954567009' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-17T07:13:10.102 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:09.576665+0000 mon.c (mon.1) 15 : audit [DBG] from='client.? 172.21.15.44:0/2954567009' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-17T07:13:10.113 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:09.576665+0000 mon.c (mon.1) 15 : audit [DBG] from='client.? 172.21.15.44:0/2954567009' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-17T07:13:10.785 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:10 smithi044 podman[40117]: 2024-09-17 07:13:10.576550887 +0000 UTC m=+9.118741487 image pull e5a616e4b9cf68dfcad7782b78e118be4310022e874d52da85c55923fb615f87 docker.io/prom/node-exporter:v0.18.1 2024-09-17T07:13:11.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:09.775294+0000 mgr.y (mgr.14144) 187 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:11.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:09.775294+0000 mgr.y (mgr.14144) 187 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:11.075 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:10 smithi044 podman[40117]: 2024-09-17 07:13:10.910384595 +0000 UTC m=+9.452575243 container create d7284955755c8b5cf23641b1955969aabd7589f2e270d493c7628f8033ad258f (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a, maintainer=The Prometheus Authors ) 2024-09-17T07:13:11.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:09.775294+0000 mgr.y (mgr.14144) 187 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:11.770 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 podman[40117]: 2024-09-17 07:13:11.532846594 +0000 UTC m=+10.075037196 container init d7284955755c8b5cf23641b1955969aabd7589f2e270d493c7628f8033ad258f (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a, maintainer=The Prometheus Authors ) 2024-09-17T07:13:11.770 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 podman[40117]: 2024-09-17 07:13:11.536623847 +0000 UTC m=+10.078814452 container start d7284955755c8b5cf23641b1955969aabd7589f2e270d493c7628f8033ad258f (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a, maintainer=The Prometheus Authors ) 2024-09-17T07:13:11.770 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[40572]: time="2024-09-17T07:13:11Z" level=info msg="Starting node_exporter (version=0.18.1, branch=HEAD, revision=3db77732e925c08f675d7404a8c46466b2ece83e)" source="node_exporter.go:156" 2024-09-17T07:13:11.770 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[40572]: time="2024-09-17T07:13:11Z" level=info msg="Build context (go=go1.12.5, user=root@b50852a1acba, date=20190604-16:41:18)" source="node_exporter.go:157" 2024-09-17T07:13:11.770 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[40572]: time="2024-09-17T07:13:11Z" level=info msg="Enabled collectors:" source="node_exporter.go:97" 2024-09-17T07:13:11.770 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[40572]: time="2024-09-17T07:13:11Z" level=info msg=" - arp" source="node_exporter.go:104" 2024-09-17T07:13:11.770 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[40572]: time="2024-09-17T07:13:11Z" level=info msg=" - bcache" source="node_exporter.go:104" 2024-09-17T07:13:11.770 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[40572]: time="2024-09-17T07:13:11Z" level=info msg=" - bonding" source="node_exporter.go:104" 2024-09-17T07:13:11.770 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[40572]: time="2024-09-17T07:13:11Z" level=info msg=" - conntrack" source="node_exporter.go:104" 2024-09-17T07:13:11.771 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[40572]: time="2024-09-17T07:13:11Z" level=info msg=" - cpu" source="node_exporter.go:104" 2024-09-17T07:13:11.771 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[40572]: time="2024-09-17T07:13:11Z" level=info msg=" - cpufreq" source="node_exporter.go:104" 2024-09-17T07:13:11.771 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[40572]: time="2024-09-17T07:13:11Z" level=info msg=" - diskstats" source="node_exporter.go:104" 2024-09-17T07:13:11.771 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[40572]: time="2024-09-17T07:13:11Z" level=info msg=" - edac" source="node_exporter.go:104" 2024-09-17T07:13:11.771 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[40572]: time="2024-09-17T07:13:11Z" level=info msg=" - entropy" source="node_exporter.go:104" 2024-09-17T07:13:11.771 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[40572]: time="2024-09-17T07:13:11Z" level=info msg=" - filefd" source="node_exporter.go:104" 2024-09-17T07:13:11.771 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[40572]: time="2024-09-17T07:13:11Z" level=info msg=" - filesystem" source="node_exporter.go:104" 2024-09-17T07:13:11.771 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[40572]: time="2024-09-17T07:13:11Z" level=info msg=" - hwmon" source="node_exporter.go:104" 2024-09-17T07:13:11.771 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[40572]: time="2024-09-17T07:13:11Z" level=info msg=" - infiniband" source="node_exporter.go:104" 2024-09-17T07:13:11.771 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[40572]: time="2024-09-17T07:13:11Z" level=info msg=" - ipvs" source="node_exporter.go:104" 2024-09-17T07:13:11.771 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[40572]: time="2024-09-17T07:13:11Z" level=info msg=" - loadavg" source="node_exporter.go:104" 2024-09-17T07:13:11.771 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[40572]: time="2024-09-17T07:13:11Z" level=info msg=" - mdadm" source="node_exporter.go:104" 2024-09-17T07:13:11.771 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[40572]: time="2024-09-17T07:13:11Z" level=info msg=" - meminfo" source="node_exporter.go:104" 2024-09-17T07:13:11.772 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[40572]: time="2024-09-17T07:13:11Z" level=info msg=" - netclass" source="node_exporter.go:104" 2024-09-17T07:13:11.772 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[40572]: time="2024-09-17T07:13:11Z" level=info msg=" - netdev" source="node_exporter.go:104" 2024-09-17T07:13:11.772 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[40572]: time="2024-09-17T07:13:11Z" level=info msg=" - netstat" source="node_exporter.go:104" 2024-09-17T07:13:11.772 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[40572]: time="2024-09-17T07:13:11Z" level=info msg=" - nfs" source="node_exporter.go:104" 2024-09-17T07:13:11.772 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[40572]: time="2024-09-17T07:13:11Z" level=info msg=" - nfsd" source="node_exporter.go:104" 2024-09-17T07:13:11.772 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[40572]: time="2024-09-17T07:13:11Z" level=info msg=" - pressure" source="node_exporter.go:104" 2024-09-17T07:13:11.772 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[40572]: time="2024-09-17T07:13:11Z" level=info msg=" - sockstat" source="node_exporter.go:104" 2024-09-17T07:13:11.772 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[40572]: time="2024-09-17T07:13:11Z" level=info msg=" - stat" source="node_exporter.go:104" 2024-09-17T07:13:11.772 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[40572]: time="2024-09-17T07:13:11Z" level=info msg=" - textfile" source="node_exporter.go:104" 2024-09-17T07:13:11.772 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[40572]: time="2024-09-17T07:13:11Z" level=info msg=" - time" source="node_exporter.go:104" 2024-09-17T07:13:11.772 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[40572]: time="2024-09-17T07:13:11Z" level=info msg=" - uname" source="node_exporter.go:104" 2024-09-17T07:13:11.772 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[40572]: time="2024-09-17T07:13:11Z" level=info msg=" - vmstat" source="node_exporter.go:104" 2024-09-17T07:13:11.772 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[40572]: time="2024-09-17T07:13:11Z" level=info msg=" - xfs" source="node_exporter.go:104" 2024-09-17T07:13:11.773 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[40572]: time="2024-09-17T07:13:11Z" level=info msg=" - zfs" source="node_exporter.go:104" 2024-09-17T07:13:11.773 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[40572]: time="2024-09-17T07:13:11Z" level=info msg="Listening on :9100" source="node_exporter.go:170" 2024-09-17T07:13:11.773 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 bash[40117]: d7284955755c8b5cf23641b1955969aabd7589f2e270d493c7628f8033ad258f 2024-09-17T07:13:11.773 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:13:11 smithi044 systemd[1]: Started Ceph node-exporter.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:13:12.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:11.523459+0000 mon.a (mon.0) 470 : audit [DBG] from='client.? 172.21.15.44:0/1613283365' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-17T07:13:12.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:11.704667+0000 mon.a (mon.0) 471 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:13:12.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:11.523459+0000 mon.a (mon.0) 470 : audit [DBG] from='client.? 172.21.15.44:0/1613283365' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-17T07:13:12.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:11.704667+0000 mon.a (mon.0) 471 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:13:12.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:11 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:11.523459+0000 mon.a (mon.0) 470 : audit [DBG] from='client.? 172.21.15.44:0/1613283365' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-17T07:13:12.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:11 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:11.704667+0000 mon.a (mon.0) 471 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:13:13.105 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:13:11.709090+0000 mgr.y (mgr.14144) 188 : cephadm [INF] Deploying daemon alertmanager.a on smithi044 2024-09-17T07:13:13.106 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:11.775556+0000 mgr.y (mgr.14144) 189 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:13.106 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:13:11.709090+0000 mgr.y (mgr.14144) 188 : cephadm [INF] Deploying daemon alertmanager.a on smithi044 2024-09-17T07:13:13.106 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:11.775556+0000 mgr.y (mgr.14144) 189 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:13.107 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388626448 got 180388626450 for osd.6 2024-09-17T07:13:13.107 DEBUG:teuthology.parallel:result is None 2024-09-17T07:13:13.113 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:12 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:13:11.709090+0000 mgr.y (mgr.14144) 188 : cephadm [INF] Deploying daemon alertmanager.a on smithi044 2024-09-17T07:13:13.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:12 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:11.775556+0000 mgr.y (mgr.14144) 189 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:13.185 INFO:teuthology.orchestra.run.smithi044.stdout:55834574885 2024-09-17T07:13:13.786 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574883 got 55834574885 for osd.1 2024-09-17T07:13:13.786 DEBUG:teuthology.parallel:result is None 2024-09-17T07:13:14.392 INFO:teuthology.orchestra.run.smithi044.stdout:206158430224 2024-09-17T07:13:14.636 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738407 got 34359738409 for osd.0 2024-09-17T07:13:14.636 DEBUG:teuthology.parallel:result is None 2024-09-17T07:13:15.102 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:13.775848+0000 mgr.y (mgr.14144) 190 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:15.102 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:14.391812+0000 mon.c (mon.1) 16 : audit [DBG] from='client.? 172.21.15.44:0/3480774544' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-17T07:13:15.103 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:13.775848+0000 mgr.y (mgr.14144) 190 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:15.103 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:14.391812+0000 mon.c (mon.1) 16 : audit [DBG] from='client.? 172.21.15.44:0/3480774544' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-17T07:13:15.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:14 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:13.775848+0000 mgr.y (mgr.14144) 190 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:15.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:14 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:14.391812+0000 mon.c (mon.1) 16 : audit [DBG] from='client.? 172.21.15.44:0/3480774544' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-17T07:13:15.614 INFO:teuthology.orchestra.run.smithi044.stdout:107374182431 2024-09-17T07:13:16.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:15.614092+0000 mon.a (mon.0) 472 : audit [DBG] from='client.? 172.21.15.44:0/3662951379' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-17T07:13:16.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:15.614092+0000 mon.a (mon.0) 472 : audit [DBG] from='client.? 172.21.15.44:0/3662951379' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-17T07:13:16.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:15.614092+0000 mon.a (mon.0) 472 : audit [DBG] from='client.? 172.21.15.44:0/3662951379' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-17T07:13:16.408 INFO:teuthology.orchestra.run.smithi044.stdout:128849018907 2024-09-17T07:13:17.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:15.776281+0000 mgr.y (mgr.14144) 191 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:17.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:16.408676+0000 mon.c (mon.1) 17 : audit [DBG] from='client.? 172.21.15.44:0/3030416275' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-17T07:13:17.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:15.776281+0000 mgr.y (mgr.14144) 191 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:17.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:16.408676+0000 mon.c (mon.1) 17 : audit [DBG] from='client.? 172.21.15.44:0/3030416275' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-17T07:13:17.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:16 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:15.776281+0000 mgr.y (mgr.14144) 191 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:17.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:16 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:16.408676+0000 mon.c (mon.1) 17 : audit [DBG] from='client.? 172.21.15.44:0/3030416275' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-17T07:13:17.190 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018903 got 128849018907 for osd.4 2024-09-17T07:13:17.191 DEBUG:teuthology.parallel:result is None 2024-09-17T07:13:18.012 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430220 got 206158430224 for osd.7 2024-09-17T07:13:18.013 DEBUG:teuthology.parallel:result is None 2024-09-17T07:13:18.571 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182427 got 107374182431 for osd.3 2024-09-17T07:13:18.571 DEBUG:teuthology.parallel:result is None 2024-09-17T07:13:19.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:18 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:17.776766+0000 mgr.y (mgr.14144) 192 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:19.191 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:17.776766+0000 mgr.y (mgr.14144) 192 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:19.192 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:17.776766+0000 mgr.y (mgr.14144) 192 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:19.750 INFO:teuthology.orchestra.run.smithi044.stdout:77309411363 2024-09-17T07:13:20.719 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:19.750702+0000 mon.a (mon.0) 473 : audit [DBG] from='client.? 172.21.15.44:0/1556448688' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-17T07:13:20.719 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:19.777026+0000 mgr.y (mgr.14144) 193 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:20.719 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:19.750702+0000 mon.a (mon.0) 473 : audit [DBG] from='client.? 172.21.15.44:0/1556448688' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-17T07:13:20.719 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:19.777026+0000 mgr.y (mgr.14144) 193 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:20.989 INFO:teuthology.orchestra.run.smithi044.stdout:154618822680 2024-09-17T07:13:21.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:19.750702+0000 mon.a (mon.0) 473 : audit [DBG] from='client.? 172.21.15.44:0/1556448688' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-17T07:13:21.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:19.777026+0000 mgr.y (mgr.14144) 193 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:21.359 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411360 got 77309411363 for osd.2 2024-09-17T07:13:21.359 DEBUG:teuthology.parallel:result is None 2024-09-17T07:13:21.951 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:21 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:20.989622+0000 mon.c (mon.1) 18 : audit [DBG] from='client.? 172.21.15.44:0/1147264212' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-17T07:13:21.952 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:21 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:20.989622+0000 mon.c (mon.1) 18 : audit [DBG] from='client.? 172.21.15.44:0/1147264212' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-17T07:13:21.980 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822675 got 154618822680 for osd.5 2024-09-17T07:13:21.980 DEBUG:teuthology.parallel:result is None 2024-09-17T07:13:21.980 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-17T07:13:21.980 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T07:13:22.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:21 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:20.989622+0000 mon.c (mon.1) 18 : audit [DBG] from='client.? 172.21.15.44:0/1147264212' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-17T07:13:22.192 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/817ae996-74c3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:13:23.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:22 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:21.777334+0000 mgr.y (mgr.14144) 194 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:23.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:22 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:21.777334+0000 mgr.y (mgr.14144) 194 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:23.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:21.777334+0000 mgr.y (mgr.14144) 194 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:24.131 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-09-17T07:13:24.132 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-09-17T07:13:24.772 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2024-09-17T07:13:23.777530+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":43156,"kb_used_data":3732,"kb_used_omap":0,"kb_used_meta":39360,"kb_avail":749918060,"statfs":{"total":767960285184,"available":767916093440,"internally_reserved":0,"allocated":3821568,"data_stored":1511015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40304640},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":"10","reported_epoch":"50","state":"active+clean","last_fresh":"2024-09-17T07:12:10.693544+0000","last_change":"2024-09-17T07:12:10.693544+0000","last_active":"2024-09-17T07:12:10.693544+0000","last_peered":"2024-09-17T07:12:10.693544+0000","last_clean":"2024-09-17T07:12:10.693544+0000","last_became_active":"2024-09-17T07:12:10.693214+0000","last_became_peered":"2024-09-17T07:12:10.693214+0000","last_unstale":"2024-09-17T07:12:10.693544+0000","last_undegraded":"2024-09-17T07:12:10.693544+0000","last_fullsized":"2024-09-17T07:12:10.693544+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:10:36.325919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:36.325919+0000","last_clean_scrub_stamp":"2024-09-17T07:10:36.325919+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430226,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5212,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93739940,"statfs":{"total":95995035648,"available":95989698560,"internally_reserved":0,"allocated":479232,"data_stored":188995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.57899999999999996}]},{"osd":1,"last update":"Tue Sep 17 07:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.53600000000000003}]},{"osd":2,"last update":"Tue Sep 17 07:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.50900000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.432}]},{"osd":4,"last update":"Tue Sep 17 07:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.40899999999999997}]},{"osd":5,"last update":"Tue Sep 17 07:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.501}]},{"osd":6,"last update":"Tue Sep 17 07:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.49299999999999999}]}]},{"osd":6,"up_from":42,"seq":180388626453,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5272,"kb_used_data":464,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93739880,"statfs":{"total":95995035648,"available":95989637120,"internally_reserved":0,"allocated":475136,"data_stored":188680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:12:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.63900000000000001}]},{"osd":1,"last update":"Tue Sep 17 07:12:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.754},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.61799999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:12:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.67200000000000004}]},{"osd":3,"last update":"Tue Sep 17 07:12:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.746}]},{"osd":4,"last update":"Tue Sep 17 07:12:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.501},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.59199999999999997}]},{"osd":5,"last update":"Tue Sep 17 07:12:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.69099999999999995}]},{"osd":7,"last update":"Tue Sep 17 07:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.70399999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5532,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739620,"statfs":{"total":95995035648,"available":95989370880,"internally_reserved":0,"allocated":479232,"data_stored":188995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:13:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.41699999999999998,"15min":0.41299999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.76,"5min":1.76,"15min":1.76},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.40000000000000002,"15min":0.38300000000000001},"min":{"1min":0.33700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":1.7110000000000001,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":0.36599999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.40699999999999997,"15min":0.40300000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.43099999999999999,"15min":0.42299999999999999},"min":{"1min":0.33100000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.443}]},{"osd":3,"last update":"Tue Sep 17 07:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.48799999999999999,"15min":0.48099999999999998},"min":{"1min":0.36899999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.4550000000000001,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47899999999999998,"15min":0.47999999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.49199999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:13:14 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.64700000000000002,"15min":0.63900000000000001},"min":{"1min":0.47599999999999998,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.63200000000000001,"15min":0.63},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.74399999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:12:31 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.63400000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:12:50 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1,"5min":1,"15min":1},"last":0.75},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.67800000000000005}]},{"osd":7,"last update":"Tue Sep 17 07:13:10 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.68899999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738413,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5592,"kb_used_data":464,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739560,"statfs":{"total":95995035648,"available":95989309440,"internally_reserved":0,"allocated":475136,"data_stored":188680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:13:22 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.36899999999999999,"15min":0.35699999999999998},"min":{"1min":0.26500000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.39100000000000001,"15min":0.36799999999999999},"min":{"1min":0.224,"5min":0.183,"15min":0.183},"max":{"1min":0.57699999999999996,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.313}]},{"osd":2,"last update":"Tue Sep 17 07:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.39100000000000001,"15min":0.39500000000000002},"min":{"1min":0.27800000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.56799999999999995,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.438,"5min":0.441,"15min":0.441},"min":{"1min":0.30399999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.32600000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:13:02 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.44700000000000001,"15min":0.443},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.49199999999999999,"15min":0.49399999999999999},"min":{"1min":0.33200000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.38100000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:13:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.58499999999999996,"15min":0.57499999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.875},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.60599999999999998,"15min":0.59799999999999998},"min":{"1min":0.41999999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.63800000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:12:31 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.61799999999999999}]},{"osd":6,"last update":"Tue Sep 17 07:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.69399999999999995},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":8.1769999999999996,"5min":8.1769999999999996,"15min":8.1769999999999996},"last":0.82499999999999996}]},{"osd":7,"last update":"Tue Sep 17 07:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.01,"15min":1.01},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":8.1899999999999995,"5min":8.1899999999999995,"15min":8.1899999999999995},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.85199999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411364,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5468,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739684,"statfs":{"total":95995035648,"available":95989436416,"internally_reserved":0,"allocated":479232,"data_stored":188995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:12:40 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.34000000000000002,"15min":0.33200000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.38500000000000001,"15min":0.379},"min":{"1min":0.26400000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.61199999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.443}]},{"osd":1,"last update":"Tue Sep 17 07:12:40 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.373,"15min":0.36199999999999999},"min":{"1min":0.28199999999999997,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.432},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.41199999999999998,"15min":0.40200000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.39100000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.45100000000000001,"15min":0.45300000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.34100000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.48199999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:13:15 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.60699999999999998,"15min":0.59899999999999998},"min":{"1min":0.53700000000000003,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.76300000000000001,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.41699999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.77900000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.65900000000000003}]},{"osd":5,"last update":"Tue Sep 17 07:12:31 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.79400000000000004}]},{"osd":6,"last update":"Tue Sep 17 07:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.746}]},{"osd":7,"last update":"Tue Sep 17 07:13:10 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.71699999999999997}]}]},{"osd":3,"up_from":25,"seq":107374182433,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5340,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739812,"statfs":{"total":95995035648,"available":95989567488,"internally_reserved":0,"allocated":479232,"data_stored":188995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:12:59 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.432,"15min":0.437},"min":{"1min":0.28899999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.57399999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.438},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.42299999999999999,"15min":0.42499999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.40999999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:12:59 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.42099999999999999,"15min":0.42399999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.46800000000000003,"15min":0.46899999999999997},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.38700000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:12:59 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.50600000000000001,"15min":0.51600000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.58699999999999997,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.46700000000000003,"15min":0.46899999999999997},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.35899999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:13:15 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.63700000000000001,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.60299999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:12:32 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.71899999999999997}]},{"osd":6,"last update":"Tue Sep 17 07:12:52 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.69299999999999995}]},{"osd":7,"last update":"Tue Sep 17 07:13:12 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.68100000000000005}]}]},{"osd":4,"up_from":30,"seq":128849018909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5404,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739748,"statfs":{"total":95995035648,"available":95989501952,"internally_reserved":0,"allocated":479232,"data_stored":188995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:13:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.56699999999999995,"15min":0.57199999999999995},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.69799999999999995,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58199999999999996,"15min":0.58599999999999997},"min":{"1min":0.45400000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.71699999999999997,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.46800000000000003}]},{"osd":1,"last update":"Tue Sep 17 07:13:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.57399999999999995,"15min":0.57899999999999996},"min":{"1min":0.41899999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.73699999999999999,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.496},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.58099999999999996,"15min":0.58399999999999996},"min":{"1min":0.40899999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.747,"5min":0.876,"15min":0.876},"last":0.41599999999999998}]},{"osd":2,"last update":"Tue Sep 17 07:13:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.57999999999999996,"15min":0.58499999999999996},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.71799999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.58499999999999996,"15min":0.59199999999999997},"min":{"1min":0.41499999999999998,"5min":0.378,"15min":0.378},"max":{"1min":0.74199999999999999,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.39100000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:13:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.56899999999999995,"15min":0.57499999999999996},"min":{"1min":0.42399999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.71099999999999997,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.58299999999999996,"15min":0.58599999999999997},"min":{"1min":0.40400000000000003,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.70199999999999996,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.40600000000000003}]},{"osd":5,"last update":"Tue Sep 17 07:12:31 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.376},{"interface":"front","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.42299999999999999}]},{"osd":6,"last update":"Tue Sep 17 07:12:53 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.51600000000000001}]},{"osd":7,"last update":"Tue Sep 17 07:13:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.44800000000000001}]}]},{"osd":5,"up_from":36,"seq":154618822681,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5336,"kb_used_data":464,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739816,"statfs":{"total":95995035648,"available":95989571584,"internally_reserved":0,"allocated":475136,"data_stored":188680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:12:32 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.439}]},{"osd":1,"last update":"Tue Sep 17 07:12:32 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.496}]},{"osd":2,"last update":"Tue Sep 17 07:12:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.42599999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:12:32 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.52400000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:12:32 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.51100000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.60799999999999998}]},{"osd":7,"last update":"Tue Sep 17 07:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.59699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:13:24.773 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T07:13:24.988 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/817ae996-74c3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:13:25.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:23.777683+0000 mgr.y (mgr.14144) 195 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:25.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:24.127698+0000 mgr.y (mgr.14144) 196 : audit [DBG] from='client.14550 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:13:25.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:23.777683+0000 mgr.y (mgr.14144) 195 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:25.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:24.127698+0000 mgr.y (mgr.14144) 196 : audit [DBG] from='client.14550 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:13:25.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:24 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:23.777683+0000 mgr.y (mgr.14144) 195 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:25.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:24 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:24.127698+0000 mgr.y (mgr.14144) 196 : audit [DBG] from='client.14550 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:13:27.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:25.778238+0000 mgr.y (mgr.14144) 197 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:27.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:25.778238+0000 mgr.y (mgr.14144) 197 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:27.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:26 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:25.778238+0000 mgr.y (mgr.14144) 197 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:27.325 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-09-17T07:13:27.327 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-09-17T07:13:28.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:27 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:27.322621+0000 mgr.y (mgr.14144) 198 : audit [DBG] from='client.24475 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:13:28.155 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:27.322621+0000 mgr.y (mgr.14144) 198 : audit [DBG] from='client.24475 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:13:28.156 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:27.322621+0000 mgr.y (mgr.14144) 198 : audit [DBG] from='client.24475 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T07:13:28.750 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":179,"stamp":"2024-09-17T07:13:25.777900+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":43156,"kb_used_data":3732,"kb_used_omap":0,"kb_used_meta":39360,"kb_avail":749918060,"statfs":{"total":767960285184,"available":767916093440,"internally_reserved":0,"allocated":3821568,"data_stored":1511015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40304640},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":"10","reported_epoch":"50","state":"active+clean","last_fresh":"2024-09-17T07:12:10.693544+0000","last_change":"2024-09-17T07:12:10.693544+0000","last_active":"2024-09-17T07:12:10.693544+0000","last_peered":"2024-09-17T07:12:10.693544+0000","last_clean":"2024-09-17T07:12:10.693544+0000","last_became_active":"2024-09-17T07:12:10.693214+0000","last_became_peered":"2024-09-17T07:12:10.693214+0000","last_unstale":"2024-09-17T07:12:10.693544+0000","last_undegraded":"2024-09-17T07:12:10.693544+0000","last_fullsized":"2024-09-17T07:12:10.693544+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T07:10:36.325919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T07:10:36.325919+0000","last_clean_scrub_stamp":"2024-09-17T07:10:36.325919+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430227,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5212,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93739940,"statfs":{"total":95995035648,"available":95989698560,"internally_reserved":0,"allocated":479232,"data_stored":188995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.52600000000000002}]},{"osd":1,"last update":"Tue Sep 17 07:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.497}]},{"osd":2,"last update":"Tue Sep 17 07:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.50600000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.47899999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.45300000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.438},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.55400000000000005}]},{"osd":6,"last update":"Tue Sep 17 07:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.51300000000000001}]}]},{"osd":6,"up_from":42,"seq":180388626454,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5272,"kb_used_data":464,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93739880,"statfs":{"total":95995035648,"available":95989637120,"internally_reserved":0,"allocated":475136,"data_stored":188680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:12:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.503},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.58099999999999996}]},{"osd":1,"last update":"Tue Sep 17 07:12:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.59399999999999997}]},{"osd":2,"last update":"Tue Sep 17 07:12:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.48099999999999998}]},{"osd":3,"last update":"Tue Sep 17 07:12:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.63200000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:12:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.371},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.60699999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:12:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.40200000000000002}]},{"osd":7,"last update":"Tue Sep 17 07:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.41799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5532,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739620,"statfs":{"total":95995035648,"available":95989370880,"internally_reserved":0,"allocated":479232,"data_stored":188995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:13:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.41699999999999998,"15min":0.41299999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.76,"5min":1.76,"15min":1.76},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.40000000000000002,"15min":0.38300000000000001},"min":{"1min":0.33700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":1.7110000000000001,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":0.36599999999999999}]},{"osd":2,"last update":"Tue Sep 17 07:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.40699999999999997,"15min":0.40300000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.43099999999999999,"15min":0.42299999999999999},"min":{"1min":0.33100000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.443}]},{"osd":3,"last update":"Tue Sep 17 07:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.48799999999999999,"15min":0.48099999999999998},"min":{"1min":0.36899999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.4550000000000001,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47899999999999998,"15min":0.47999999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.49199999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:13:14 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.64700000000000002,"15min":0.63900000000000001},"min":{"1min":0.47599999999999998,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.63200000000000001,"15min":0.63},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.74399999999999999}]},{"osd":5,"last update":"Tue Sep 17 07:12:31 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.63400000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:12:50 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1,"5min":1,"15min":1},"last":0.75},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.67800000000000005}]},{"osd":7,"last update":"Tue Sep 17 07:13:10 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.68899999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738413,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5592,"kb_used_data":464,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739560,"statfs":{"total":95995035648,"available":95989309440,"internally_reserved":0,"allocated":475136,"data_stored":188680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 07:13:22 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.36899999999999999,"15min":0.35699999999999998},"min":{"1min":0.26500000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.39100000000000001,"15min":0.36799999999999999},"min":{"1min":0.224,"5min":0.183,"15min":0.183},"max":{"1min":0.57699999999999996,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.313}]},{"osd":2,"last update":"Tue Sep 17 07:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.39100000000000001,"15min":0.39500000000000002},"min":{"1min":0.27800000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.56799999999999995,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.438,"5min":0.441,"15min":0.441},"min":{"1min":0.30399999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.32600000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:13:02 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.44700000000000001,"15min":0.443},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.49199999999999999,"15min":0.49399999999999999},"min":{"1min":0.33200000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.38100000000000001}]},{"osd":4,"last update":"Tue Sep 17 07:13:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.58499999999999996,"15min":0.57499999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.875},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.60599999999999998,"15min":0.59799999999999998},"min":{"1min":0.41999999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.63800000000000001}]},{"osd":5,"last update":"Tue Sep 17 07:12:31 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.61799999999999999}]},{"osd":6,"last update":"Tue Sep 17 07:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.69399999999999995},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":8.1769999999999996,"5min":8.1769999999999996,"15min":8.1769999999999996},"last":0.82499999999999996}]},{"osd":7,"last update":"Tue Sep 17 07:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.01,"15min":1.01},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":8.1899999999999995,"5min":8.1899999999999995,"15min":8.1899999999999995},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.85199999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411364,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5468,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739684,"statfs":{"total":95995035648,"available":95989436416,"internally_reserved":0,"allocated":479232,"data_stored":188995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:12:40 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.34000000000000002,"15min":0.33200000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.38500000000000001,"15min":0.379},"min":{"1min":0.26400000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.61199999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.443}]},{"osd":1,"last update":"Tue Sep 17 07:12:40 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.373,"15min":0.36199999999999999},"min":{"1min":0.28199999999999997,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.432},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.41199999999999998,"15min":0.40200000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.39100000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.45100000000000001,"15min":0.45300000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.34100000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.48199999999999998}]},{"osd":4,"last update":"Tue Sep 17 07:13:15 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.60699999999999998,"15min":0.59899999999999998},"min":{"1min":0.53700000000000003,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.76300000000000001,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.41699999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.77900000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.65900000000000003}]},{"osd":5,"last update":"Tue Sep 17 07:12:31 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.79400000000000004}]},{"osd":6,"last update":"Tue Sep 17 07:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.746}]},{"osd":7,"last update":"Tue Sep 17 07:13:10 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.71699999999999997}]}]},{"osd":3,"up_from":25,"seq":107374182433,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5340,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739812,"statfs":{"total":95995035648,"available":95989567488,"internally_reserved":0,"allocated":479232,"data_stored":188995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:12:59 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.432,"15min":0.437},"min":{"1min":0.28899999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.57399999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.438},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.42299999999999999,"15min":0.42499999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.40999999999999998}]},{"osd":1,"last update":"Tue Sep 17 07:12:59 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.42099999999999999,"15min":0.42399999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.46800000000000003,"15min":0.46899999999999997},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.38700000000000001}]},{"osd":2,"last update":"Tue Sep 17 07:12:59 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.50600000000000001,"15min":0.51600000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.58699999999999997,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.46700000000000003,"15min":0.46899999999999997},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.35899999999999999}]},{"osd":4,"last update":"Tue Sep 17 07:13:15 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.63700000000000001,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.60299999999999998}]},{"osd":5,"last update":"Tue Sep 17 07:12:32 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.71899999999999997}]},{"osd":6,"last update":"Tue Sep 17 07:12:52 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.69299999999999995}]},{"osd":7,"last update":"Tue Sep 17 07:13:12 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.68100000000000005}]}]},{"osd":4,"up_from":30,"seq":128849018909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5404,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739748,"statfs":{"total":95995035648,"available":95989501952,"internally_reserved":0,"allocated":479232,"data_stored":188995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:13:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.56699999999999995,"15min":0.57199999999999995},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.69799999999999995,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58199999999999996,"15min":0.58599999999999997},"min":{"1min":0.45400000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.71699999999999997,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.46800000000000003}]},{"osd":1,"last update":"Tue Sep 17 07:13:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.57399999999999995,"15min":0.57899999999999996},"min":{"1min":0.41899999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.73699999999999999,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.496},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.58099999999999996,"15min":0.58399999999999996},"min":{"1min":0.40899999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.747,"5min":0.876,"15min":0.876},"last":0.41599999999999998}]},{"osd":2,"last update":"Tue Sep 17 07:13:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.57999999999999996,"15min":0.58499999999999996},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.71799999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.58499999999999996,"15min":0.59199999999999997},"min":{"1min":0.41499999999999998,"5min":0.378,"15min":0.378},"max":{"1min":0.74199999999999999,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.39100000000000001}]},{"osd":3,"last update":"Tue Sep 17 07:13:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.56899999999999995,"15min":0.57499999999999996},"min":{"1min":0.42399999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.71099999999999997,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.58299999999999996,"15min":0.58599999999999997},"min":{"1min":0.40400000000000003,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.70199999999999996,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.40600000000000003}]},{"osd":5,"last update":"Tue Sep 17 07:12:31 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.376},{"interface":"front","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.42299999999999999}]},{"osd":6,"last update":"Tue Sep 17 07:12:53 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.51600000000000001}]},{"osd":7,"last update":"Tue Sep 17 07:13:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.44800000000000001}]}]},{"osd":5,"up_from":36,"seq":154618822681,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5336,"kb_used_data":464,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739816,"statfs":{"total":95995035648,"available":95989571584,"internally_reserved":0,"allocated":475136,"data_stored":188680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 07:12:32 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.439}]},{"osd":1,"last update":"Tue Sep 17 07:12:32 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.496}]},{"osd":2,"last update":"Tue Sep 17 07:12:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.42599999999999999}]},{"osd":3,"last update":"Tue Sep 17 07:12:32 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.52400000000000002}]},{"osd":4,"last update":"Tue Sep 17 07:12:32 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.51100000000000001}]},{"osd":6,"last update":"Tue Sep 17 07:12:51 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.60799999999999998}]},{"osd":7,"last update":"Tue Sep 17 07:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.59699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T07:13:28.751 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-17T07:13:28.751 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-17T07:13:28.751 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-17T07:13:28.752 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-17T07:13:28.996 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/817ae996-74c3-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T07:13:29.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:28 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:27.778702+0000 mgr.y (mgr.14144) 199 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:29.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:28 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:27.778702+0000 mgr.y (mgr.14144) 199 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:29.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:28 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:27.778702+0000 mgr.y (mgr.14144) 199 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:30.916 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:30 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:29.779005+0000 mgr.y (mgr.14144) 200 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:31.000 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:29.779005+0000 mgr.y (mgr.14144) 200 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:31.000 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:29.779005+0000 mgr.y (mgr.14144) 200 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:32.673 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-09-17T07:13:33.108 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:32 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:31.779298+0000 mgr.y (mgr.14144) 201 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:33.108 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:32 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:32.671243+0000 mon.a (mon.0) 474 : audit [DBG] from='client.? 172.21.15.44:0/3422892441' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-17T07:13:33.108 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:32 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:31.779298+0000 mgr.y (mgr.14144) 201 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:33.108 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:32 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:32.671243+0000 mon.a (mon.0) 474 : audit [DBG] from='client.? 172.21.15.44:0/3422892441' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-17T07:13:33.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:31.779298+0000 mgr.y (mgr.14144) 201 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:33.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:32.671243+0000 mon.a (mon.0) 474 : audit [DBG] from='client.? 172.21.15.44:0/3422892441' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-17T07:13:34.325 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:13:34 smithi044 systemd[1]: Starting Ceph alertmanager.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:13:34.502 INFO:teuthology.orchestra.run.smithi044.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-17T07:13:34.502 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-17T07:13:34.502 INFO:tasks.cephadm:Setup complete, yielding 2024-09-17T07:13:34.502 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-17T07:13:34.515 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi044.front.sepia.ceph.com 2024-09-17T07:13:34.516 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-09-17T07:13:34.793 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:13:34 smithi044 podman[41692]: 2024-09-17 07:13:34.56413826 +0000 UTC m=+0.018522200 image pull 0881eb8f169f5556a292b4e2c01d683172b12830a62a9225a98a8e206bb734f0 docker.io/prom/alertmanager:v0.20.0 2024-09-17T07:13:34.793 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:13:34 smithi044 podman[41692]: 2024-09-17 07:13:34.675376456 +0000 UTC m=+0.129760384 volume create 3fbf3caceb635d2d9b9e7f10229d5e0b581e13828c5dc3fcb974004172ee87af 2024-09-17T07:13:34.793 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:13:34 smithi044 podman[41692]: 2024-09-17 07:13:34.792597145 +0000 UTC m=+0.246981072 container create f43892f6f8a5af635fef1df0d7aa55f2253b4ec36c9d04392f017fa2a3596f02 (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a, maintainer=The Prometheus Authors ) 2024-09-17T07:13:35.053 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:33.779600+0000 mgr.y (mgr.14144) 202 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:35.054 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:33.779600+0000 mgr.y (mgr.14144) 202 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:35.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:33.779600+0000 mgr.y (mgr.14144) 202 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:35.575 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:13:35 smithi044 podman[41692]: 2024-09-17 07:13:35.314299966 +0000 UTC m=+0.768683896 container init f43892f6f8a5af635fef1df0d7aa55f2253b4ec36c9d04392f017fa2a3596f02 (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a, maintainer=The Prometheus Authors ) 2024-09-17T07:13:35.575 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:13:35 smithi044 podman[41692]: 2024-09-17 07:13:35.317317559 +0000 UTC m=+0.771701487 container start f43892f6f8a5af635fef1df0d7aa55f2253b4ec36c9d04392f017fa2a3596f02 (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a, maintainer=The Prometheus Authors ) 2024-09-17T07:13:35.575 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:13:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a[41732]: level=info ts=2024-09-17T07:13:35.337Z caller=main.go:231 msg="Starting Alertmanager" version="(version=0.20.0, branch=HEAD, revision=f74be0400a6243d10bb53812d6fa408ad71ff32d)" 2024-09-17T07:13:35.575 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:13:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a[41732]: level=info ts=2024-09-17T07:13:35.337Z caller=main.go:232 build_context="(go=go1.13.5, user=root@00c3106655f8, date=20191211-14:13:14)" 2024-09-17T07:13:35.575 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:13:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a[41732]: level=info ts=2024-09-17T07:13:35.338Z caller=cluster.go:161 component=cluster msg="setting advertise address explicitly" addr=172.21.15.44 port=9094 2024-09-17T07:13:35.575 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:13:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a[41732]: level=info ts=2024-09-17T07:13:35.339Z caller=cluster.go:623 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-17T07:13:35.575 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:13:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a[41732]: level=info ts=2024-09-17T07:13:35.358Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-17T07:13:35.575 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:13:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a[41732]: level=info ts=2024-09-17T07:13:35.358Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-17T07:13:35.575 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:13:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a[41732]: level=info ts=2024-09-17T07:13:35.361Z caller=main.go:497 msg=Listening address=:9093 2024-09-17T07:13:35.575 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:13:35 smithi044 bash[41692]: f43892f6f8a5af635fef1df0d7aa55f2253b4ec36c9d04392f017fa2a3596f02 2024-09-17T07:13:35.576 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:13:35 smithi044 systemd[1]: Started Ceph alertmanager.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:13:36.637 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:35.474898+0000 mon.a (mon.0) 475 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:13:36.637 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:35.629696+0000 mon.a (mon.0) 476 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:13:36.637 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:35.634831+0000 mon.a (mon.0) 477 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:13:36.638 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:35.635315+0000 mon.a (mon.0) 478 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T07:13:36.638 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:35.641133+0000 mon.a (mon.0) 479 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:13:36.638 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:35.474898+0000 mon.a (mon.0) 475 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:13:36.638 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:35.629696+0000 mon.a (mon.0) 476 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:13:36.638 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:35.634831+0000 mon.a (mon.0) 477 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:13:36.638 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:35.635315+0000 mon.a (mon.0) 478 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T07:13:36.638 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:35.641133+0000 mon.a (mon.0) 479 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:13:36.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:35.474898+0000 mon.a (mon.0) 475 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:13:36.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:35.629696+0000 mon.a (mon.0) 476 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:13:36.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:35.634831+0000 mon.a (mon.0) 477 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:13:36.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:35.635315+0000 mon.a (mon.0) 478 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T07:13:36.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:35.641133+0000 mon.a (mon.0) 479 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:13:37.824 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:13:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a[41732]: level=info ts=2024-09-17T07:13:37.339Z caller=cluster.go:648 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000102052s 2024-09-17T07:13:37.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:35.635769+0000 mgr.y (mgr.14144) 203 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T07:13:37.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:13:35.644926+0000 mgr.y (mgr.14144) 204 : cephadm [INF] Deploying daemon grafana.a on smithi179 2024-09-17T07:13:37.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:35.779879+0000 mgr.y (mgr.14144) 205 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:37.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:35.635769+0000 mgr.y (mgr.14144) 203 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T07:13:37.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:13:35.644926+0000 mgr.y (mgr.14144) 204 : cephadm [INF] Deploying daemon grafana.a on smithi179 2024-09-17T07:13:37.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:35.779879+0000 mgr.y (mgr.14144) 205 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:37.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:37 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:35.635769+0000 mgr.y (mgr.14144) 203 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T07:13:37.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:37 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:13:35.644926+0000 mgr.y (mgr.14144) 204 : cephadm [INF] Deploying daemon grafana.a on smithi179 2024-09-17T07:13:37.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:37 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:35.779879+0000 mgr.y (mgr.14144) 205 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:38.683 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:13:38.684 INFO:teuthology.orchestra.run.smithi044.stdout: "id": "e4999de9-4607-4aa1-95cb-dde6f41e8c30", 2024-09-17T07:13:38.684 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "r", 2024-09-17T07:13:38.684 INFO:teuthology.orchestra.run.smithi044.stdout: "current_period": "00b9c429-3f2a-474a-8f09-956d28bc0a36", 2024-09-17T07:13:38.684 INFO:teuthology.orchestra.run.smithi044.stdout: "epoch": 1 2024-09-17T07:13:38.685 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:13:39.045 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:37.646479+0000 mon.a (mon.0) 480 : cluster [DBG] osdmap e51: 8 total, 8 up, 8 in 2024-09-17T07:13:39.045 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:37.647883+0000 mon.c (mon.1) 19 : audit [INF] from='client.? 172.21.15.44:0/1653788739' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-17T07:13:39.045 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:37.648889+0000 mon.a (mon.0) 481 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-17T07:13:39.045 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:37.780184+0000 mgr.y (mgr.14144) 206 : cluster [DBG] pgmap v186: 33 pgs: 32 unknown, 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:39.045 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:37.646479+0000 mon.a (mon.0) 480 : cluster [DBG] osdmap e51: 8 total, 8 up, 8 in 2024-09-17T07:13:39.045 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:37.647883+0000 mon.c (mon.1) 19 : audit [INF] from='client.? 172.21.15.44:0/1653788739' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-17T07:13:39.046 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:37.648889+0000 mon.a (mon.0) 481 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-17T07:13:39.046 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:37.780184+0000 mgr.y (mgr.14144) 206 : cluster [DBG] pgmap v186: 33 pgs: 32 unknown, 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:39.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:38 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:37.646479+0000 mon.a (mon.0) 480 : cluster [DBG] osdmap e51: 8 total, 8 up, 8 in 2024-09-17T07:13:39.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:38 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:37.647883+0000 mon.c (mon.1) 19 : audit [INF] from='client.? 172.21.15.44:0/1653788739' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-17T07:13:39.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:38 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:37.648889+0000 mon.a (mon.0) 481 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-17T07:13:39.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:38 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:37.780184+0000 mgr.y (mgr.14144) 206 : cluster [DBG] pgmap v186: 33 pgs: 32 unknown, 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:39.785 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-09-17T07:13:40.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:38.652829+0000 mon.a (mon.0) 482 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-17T07:13:40.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:38.653102+0000 mon.a (mon.0) 483 : cluster [DBG] osdmap e52: 8 total, 8 up, 8 in 2024-09-17T07:13:40.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:38.652829+0000 mon.a (mon.0) 482 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-17T07:13:40.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:38.653102+0000 mon.a (mon.0) 483 : cluster [DBG] osdmap e52: 8 total, 8 up, 8 in 2024-09-17T07:13:40.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:38.652829+0000 mon.a (mon.0) 482 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-17T07:13:40.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:38.653102+0000 mon.a (mon.0) 483 : cluster [DBG] osdmap e52: 8 total, 8 up, 8 in 2024-09-17T07:13:41.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:39.665832+0000 mon.a (mon.0) 484 : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2024-09-17T07:13:41.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:39.780562+0000 mgr.y (mgr.14144) 207 : cluster [DBG] pgmap v189: 33 pgs: 2 creating+peering, 1 active+clean, 30 unknown; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:41.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:39.665832+0000 mon.a (mon.0) 484 : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2024-09-17T07:13:41.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:39.780562+0000 mgr.y (mgr.14144) 207 : cluster [DBG] pgmap v189: 33 pgs: 2 creating+peering, 1 active+clean, 30 unknown; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:41.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:40 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:39.665832+0000 mon.a (mon.0) 484 : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2024-09-17T07:13:41.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:40 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:39.780562+0000 mgr.y (mgr.14144) 207 : cluster [DBG] pgmap v189: 33 pgs: 2 creating+peering, 1 active+clean, 30 unknown; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:41.674 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:13:41.674 INFO:teuthology.orchestra.run.smithi044.stdout: "id": "2e99a19f-76a1-4791-939d-7bc9089c963b", 2024-09-17T07:13:41.674 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "default", 2024-09-17T07:13:41.674 INFO:teuthology.orchestra.run.smithi044.stdout: "api_name": "default", 2024-09-17T07:13:41.674 INFO:teuthology.orchestra.run.smithi044.stdout: "is_master": "true", 2024-09-17T07:13:41.674 INFO:teuthology.orchestra.run.smithi044.stdout: "endpoints": [], 2024-09-17T07:13:41.674 INFO:teuthology.orchestra.run.smithi044.stdout: "hostnames": [], 2024-09-17T07:13:41.674 INFO:teuthology.orchestra.run.smithi044.stdout: "hostnames_s3website": [], 2024-09-17T07:13:41.675 INFO:teuthology.orchestra.run.smithi044.stdout: "master_zone": "", 2024-09-17T07:13:41.675 INFO:teuthology.orchestra.run.smithi044.stdout: "zones": [], 2024-09-17T07:13:41.675 INFO:teuthology.orchestra.run.smithi044.stdout: "placement_targets": [], 2024-09-17T07:13:41.675 INFO:teuthology.orchestra.run.smithi044.stdout: "default_placement": "", 2024-09-17T07:13:41.675 INFO:teuthology.orchestra.run.smithi044.stdout: "realm_id": "e4999de9-4607-4aa1-95cb-dde6f41e8c30", 2024-09-17T07:13:41.675 INFO:teuthology.orchestra.run.smithi044.stdout: "sync_policy": { 2024-09-17T07:13:41.675 INFO:teuthology.orchestra.run.smithi044.stdout: "groups": [] 2024-09-17T07:13:41.675 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-09-17T07:13:41.675 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:13:42.500 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-09-17T07:13:43.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:41.781162+0000 mgr.y (mgr.14144) 208 : cluster [DBG] pgmap v190: 33 pgs: 30 active+clean, 2 creating+peering, 1 unknown; 516 B data, 43 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:43.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:41.781162+0000 mgr.y (mgr.14144) 208 : cluster [DBG] pgmap v190: 33 pgs: 30 active+clean, 2 creating+peering, 1 unknown; 516 B data, 43 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:43.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:41.781162+0000 mgr.y (mgr.14144) 208 : cluster [DBG] pgmap v190: 33 pgs: 30 active+clean, 2 creating+peering, 1 unknown; 516 B data, 43 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:13:44.634 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:13:44.634 INFO:teuthology.orchestra.run.smithi044.stdout: "id": "86dcd634-218d-4e94-8f3e-3feca791c62c", 2024-09-17T07:13:44.634 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "z", 2024-09-17T07:13:44.635 INFO:teuthology.orchestra.run.smithi044.stdout: "domain_root": "z.rgw.meta:root", 2024-09-17T07:13:44.635 INFO:teuthology.orchestra.run.smithi044.stdout: "control_pool": "z.rgw.control", 2024-09-17T07:13:44.635 INFO:teuthology.orchestra.run.smithi044.stdout: "gc_pool": "z.rgw.log:gc", 2024-09-17T07:13:44.635 INFO:teuthology.orchestra.run.smithi044.stdout: "lc_pool": "z.rgw.log:lc", 2024-09-17T07:13:44.635 INFO:teuthology.orchestra.run.smithi044.stdout: "log_pool": "z.rgw.log", 2024-09-17T07:13:44.635 INFO:teuthology.orchestra.run.smithi044.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-09-17T07:13:44.635 INFO:teuthology.orchestra.run.smithi044.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-09-17T07:13:44.636 INFO:teuthology.orchestra.run.smithi044.stdout: "roles_pool": "z.rgw.meta:roles", 2024-09-17T07:13:44.636 INFO:teuthology.orchestra.run.smithi044.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-09-17T07:13:44.636 INFO:teuthology.orchestra.run.smithi044.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-09-17T07:13:44.636 INFO:teuthology.orchestra.run.smithi044.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-09-17T07:13:44.636 INFO:teuthology.orchestra.run.smithi044.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-09-17T07:13:44.636 INFO:teuthology.orchestra.run.smithi044.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-09-17T07:13:44.636 INFO:teuthology.orchestra.run.smithi044.stdout: "otp_pool": "z.rgw.otp", 2024-09-17T07:13:44.636 INFO:teuthology.orchestra.run.smithi044.stdout: "system_key": { 2024-09-17T07:13:44.636 INFO:teuthology.orchestra.run.smithi044.stdout: "access_key": "", 2024-09-17T07:13:44.636 INFO:teuthology.orchestra.run.smithi044.stdout: "secret_key": "" 2024-09-17T07:13:44.636 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:13:44.636 INFO:teuthology.orchestra.run.smithi044.stdout: "placement_pools": [ 2024-09-17T07:13:44.636 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-09-17T07:13:44.637 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "default-placement", 2024-09-17T07:13:44.637 INFO:teuthology.orchestra.run.smithi044.stdout: "val": { 2024-09-17T07:13:44.640 INFO:teuthology.orchestra.run.smithi044.stdout: "index_pool": "z.rgw.buckets.index", 2024-09-17T07:13:44.640 INFO:teuthology.orchestra.run.smithi044.stdout: "storage_classes": { 2024-09-17T07:13:44.641 INFO:teuthology.orchestra.run.smithi044.stdout: "STANDARD": { 2024-09-17T07:13:44.641 INFO:teuthology.orchestra.run.smithi044.stdout: "data_pool": "z.rgw.buckets.data" 2024-09-17T07:13:44.641 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-09-17T07:13:44.641 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:13:44.641 INFO:teuthology.orchestra.run.smithi044.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-09-17T07:13:44.641 INFO:teuthology.orchestra.run.smithi044.stdout: "index_type": 0 2024-09-17T07:13:44.641 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-09-17T07:13:44.641 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-09-17T07:13:44.642 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-09-17T07:13:44.642 INFO:teuthology.orchestra.run.smithi044.stdout: "realm_id": "e4999de9-4607-4aa1-95cb-dde6f41e8c30", 2024-09-17T07:13:44.642 INFO:teuthology.orchestra.run.smithi044.stdout: "notif_pool": "z.rgw.log:notif" 2024-09-17T07:13:44.642 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:13:45.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:43.781502+0000 mgr.y (mgr.14144) 209 : cluster [DBG] pgmap v191: 33 pgs: 33 active+clean; 562 B data, 43 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-09-17T07:13:45.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:43.781502+0000 mgr.y (mgr.14144) 209 : cluster [DBG] pgmap v191: 33 pgs: 33 active+clean; 562 B data, 43 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-09-17T07:13:45.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:43.781502+0000 mgr.y (mgr.14144) 209 : cluster [DBG] pgmap v191: 33 pgs: 33 active+clean; 562 B data, 43 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-09-17T07:13:45.496 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-09-17T07:13:45.574 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:13:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a[41732]: level=info ts=2024-09-17T07:13:45.340Z caller=cluster.go:640 component=cluster msg="gossip settled; proceeding" elapsed=10.000609739s 2024-09-17T07:13:47.057 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:45.781994+0000 mgr.y (mgr.14144) 210 : cluster [DBG] pgmap v192: 33 pgs: 33 active+clean; 1.7 KiB data, 43 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-17T07:13:47.057 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:45.781994+0000 mgr.y (mgr.14144) 210 : cluster [DBG] pgmap v192: 33 pgs: 33 active+clean; 1.7 KiB data, 43 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-17T07:13:47.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:45.781994+0000 mgr.y (mgr.14144) 210 : cluster [DBG] pgmap v192: 33 pgs: 33 active+clean; 1.7 KiB data, 43 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-17T07:13:48.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:47.742454+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]: dispatch 2024-09-17T07:13:48.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:47.742454+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]: dispatch 2024-09-17T07:13:48.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:47.742454+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]: dispatch 2024-09-17T07:13:49.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:47.782524+0000 mgr.y (mgr.14144) 211 : cluster [DBG] pgmap v193: 33 pgs: 33 active+clean; 1.8 KiB data, 43 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1.3 KiB/s wr, 2 op/s 2024-09-17T07:13:49.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:47.847075+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]': finished 2024-09-17T07:13:49.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:47.847150+0000 mon.a (mon.0) 487 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2024-09-17T07:13:49.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:47.850300+0000 mon.c (mon.1) 20 : audit [INF] from='client.? 172.21.15.44:0/1084728869' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-09-17T07:13:49.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:47.851338+0000 mon.a (mon.0) 488 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-09-17T07:13:49.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:47.868941+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:13:49.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:47.869958+0000 mon.a (mon.0) 490 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:13:49.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:47.782524+0000 mgr.y (mgr.14144) 211 : cluster [DBG] pgmap v193: 33 pgs: 33 active+clean; 1.8 KiB data, 43 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1.3 KiB/s wr, 2 op/s 2024-09-17T07:13:49.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:47.847075+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]': finished 2024-09-17T07:13:49.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:47.847150+0000 mon.a (mon.0) 487 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2024-09-17T07:13:49.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:47.850300+0000 mon.c (mon.1) 20 : audit [INF] from='client.? 172.21.15.44:0/1084728869' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-09-17T07:13:49.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:47.851338+0000 mon.a (mon.0) 488 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-09-17T07:13:49.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:47.868941+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:13:49.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:47.869958+0000 mon.a (mon.0) 490 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:13:49.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:47.782524+0000 mgr.y (mgr.14144) 211 : cluster [DBG] pgmap v193: 33 pgs: 33 active+clean; 1.8 KiB data, 43 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1.3 KiB/s wr, 2 op/s 2024-09-17T07:13:49.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:47.847075+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]': finished 2024-09-17T07:13:49.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:47.847150+0000 mon.a (mon.0) 487 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2024-09-17T07:13:49.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:47.850300+0000 mon.c (mon.1) 20 : audit [INF] from='client.? 172.21.15.44:0/1084728869' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-09-17T07:13:49.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:47.851338+0000 mon.a (mon.0) 488 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-09-17T07:13:49.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:47.868941+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:13:49.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:47.869958+0000 mon.a (mon.0) 490 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:13:49.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:48.850356+0000 mon.a (mon.0) 491 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-09-17T07:13:49.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:48.850632+0000 mon.a (mon.0) 492 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2024-09-17T07:13:50.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:48.850356+0000 mon.a (mon.0) 491 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-09-17T07:13:50.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:48.850632+0000 mon.a (mon.0) 492 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2024-09-17T07:13:50.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:48.850356+0000 mon.a (mon.0) 491 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-09-17T07:13:50.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:48.850632+0000 mon.a (mon.0) 492 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2024-09-17T07:13:50.864 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:50 smithi179 systemd[1]: Starting Ceph grafana.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:13:51.144 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:49.782958+0000 mgr.y (mgr.14144) 212 : cluster [DBG] pgmap v196: 65 pgs: 6 creating+peering, 26 unknown, 33 active+clean; 1.8 KiB data, 43 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 1023 B/s wr, 1 op/s 2024-09-17T07:13:51.144 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:49.858566+0000 mon.a (mon.0) 493 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2024-09-17T07:13:51.144 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:49.860692+0000 mon.c (mon.1) 21 : audit [INF] from='client.? 172.21.15.44:0/1084728869' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-09-17T07:13:51.144 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:49.867161+0000 mon.a (mon.0) 494 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-09-17T07:13:51.145 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:50 smithi179 podman[36538]: 2024-09-17 07:13:50.801576271 +0000 UTC m=+0.016077590 image pull 557c83e11646f123a27b5e4b62ac6c45e7bb8b2e90d6044034d0db5b7019415c docker.io/ceph/ceph-grafana:6.7.4 2024-09-17T07:13:51.145 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:50 smithi179 podman[36538]: 2024-09-17 07:13:50.900429424 +0000 UTC m=+0.114930742 container create 2436fcd841b8e31657c32d9448424d95e0f4c57548d9c3f03bef30d53eb002f6 (image=docker.io/ceph/ceph-grafana:6.7.4, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a, org.label-schema.build-date=20201204, summary=Grafana Container configured for Ceph mgr/dashboard integration, org.label-schema.schema-version=1.0, io.buildah.version=1.19.8, maintainer=Paul Cuzner , description=Ceph Grafana Container, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Base Image, org.label-schema.license=GPLv2) 2024-09-17T07:13:51.145 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 podman[36538]: 2024-09-17 07:13:51.144366619 +0000 UTC m=+0.358867941 container init 2436fcd841b8e31657c32d9448424d95e0f4c57548d9c3f03bef30d53eb002f6 (image=docker.io/ceph/ceph-grafana:6.7.4, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a, summary=Grafana Container configured for Ceph mgr/dashboard integration, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Base Image, description=Ceph Grafana Container, org.label-schema.build-date=20201204, org.label-schema.vendor=CentOS, io.buildah.version=1.19.8, maintainer=Paul Cuzner , org.label-schema.license=GPLv2) 2024-09-17T07:13:51.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:49.782958+0000 mgr.y (mgr.14144) 212 : cluster [DBG] pgmap v196: 65 pgs: 6 creating+peering, 26 unknown, 33 active+clean; 1.8 KiB data, 43 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 1023 B/s wr, 1 op/s 2024-09-17T07:13:51.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:49.858566+0000 mon.a (mon.0) 493 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2024-09-17T07:13:51.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:49.860692+0000 mon.c (mon.1) 21 : audit [INF] from='client.? 172.21.15.44:0/1084728869' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-09-17T07:13:51.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:49.867161+0000 mon.a (mon.0) 494 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-09-17T07:13:51.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:49.782958+0000 mgr.y (mgr.14144) 212 : cluster [DBG] pgmap v196: 65 pgs: 6 creating+peering, 26 unknown, 33 active+clean; 1.8 KiB data, 43 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 1023 B/s wr, 1 op/s 2024-09-17T07:13:51.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:49.858566+0000 mon.a (mon.0) 493 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2024-09-17T07:13:51.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:49.860692+0000 mon.c (mon.1) 21 : audit [INF] from='client.? 172.21.15.44:0/1084728869' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-09-17T07:13:51.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:49.867161+0000 mon.a (mon.0) 494 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-09-17T07:13:51.614 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 podman[36538]: 2024-09-17 07:13:51.147200637 +0000 UTC m=+0.361701956 container start 2436fcd841b8e31657c32d9448424d95e0f4c57548d9c3f03bef30d53eb002f6 (image=docker.io/ceph/ceph-grafana:6.7.4, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a, description=Ceph Grafana Container, io.buildah.version=1.19.8, summary=Grafana Container configured for Ceph mgr/dashboard integration, org.label-schema.schema-version=1.0, maintainer=Paul Cuzner , org.label-schema.build-date=20201204, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image) 2024-09-17T07:13:51.615 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Starting Grafana" logger=server version=6.7.4 commit=8e44bbc5f5 branch=HEAD compiled=2020-05-26T17:35:38+0000 2024-09-17T07:13:51.615 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-09-17T07:13:51.615 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-09-17T07:13:51.615 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-17T07:13:51.615 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-17T07:13:51.615 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-17T07:13:51.615 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-17T07:13:51.615 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-09-17T07:13:51.615 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-09-17T07:13:51.615 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-09-17T07:13:51.615 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-09-17T07:13:51.616 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-09-17T07:13:51.616 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="App mode production" logger=settings 2024-09-17T07:13:51.616 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Initializing SqlStore" logger=server 2024-09-17T07:13:51.616 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-09-17T07:13:51.616 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Starting DB migration" logger=migrator 2024-09-17T07:13:51.616 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2024-09-17T07:13:51.616 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2024-09-17T07:13:51.616 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2024-09-17T07:13:51.616 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2024-09-17T07:13:51.616 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2024-09-17T07:13:51.616 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2024-09-17T07:13:51.616 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2024-09-17T07:13:51.617 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2024-09-17T07:13:51.617 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2024-09-17T07:13:51.617 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2024-09-17T07:13:51.617 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-09-17T07:13:51.617 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2024-09-17T07:13:51.617 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2024-09-17T07:13:51.617 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2024-09-17T07:13:51.617 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2024-09-17T07:13:51.617 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2024-09-17T07:13:51.617 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2024-09-17T07:13:51.617 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2024-09-17T07:13:51.617 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2024-09-17T07:13:51.617 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2024-09-17T07:13:51.617 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2024-09-17T07:13:51.618 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-09-17T07:13:51.619 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-09-17T07:13:51.619 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-09-17T07:13:51.619 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-09-17T07:13:51.619 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-09-17T07:13:51.619 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-09-17T07:13:51.620 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-09-17T07:13:51.620 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-09-17T07:13:51.620 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-09-17T07:13:51.620 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-09-17T07:13:51.620 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-09-17T07:13:51.620 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-09-17T07:13:51.620 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-09-17T07:13:51.620 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-09-17T07:13:51.620 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-09-17T07:13:51.620 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-09-17T07:13:51.620 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-09-17T07:13:51.620 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-17T07:13:51.621 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-09-17T07:13:51.621 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-09-17T07:13:51.621 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-09-17T07:13:51.621 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-09-17T07:13:51.621 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-09-17T07:13:51.621 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-09-17T07:13:51.621 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-09-17T07:13:51.621 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-09-17T07:13:51.621 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-09-17T07:13:51.621 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-09-17T07:13:51.621 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-09-17T07:13:51.622 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-09-17T07:13:51.622 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-09-17T07:13:51.622 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-09-17T07:13:51.622 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-09-17T07:13:51.622 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-09-17T07:13:51.622 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-09-17T07:13:51.622 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-09-17T07:13:51.622 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-09-17T07:13:51.622 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-09-17T07:13:51.622 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-09-17T07:13:51.622 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-09-17T07:13:51.622 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-09-17T07:13:51.622 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-09-17T07:13:51.623 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-17T07:13:51.623 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-09-17T07:13:51.623 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-17T07:13:51.623 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-09-17T07:13:51.623 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-17T07:13:51.623 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-17T07:13:51.623 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-09-17T07:13:51.623 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-09-17T07:13:51.623 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-09-17T07:13:51.624 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-09-17T07:13:51.624 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-09-17T07:13:51.624 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-09-17T07:13:51.624 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-09-17T07:13:51.624 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-09-17T07:13:51.624 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-09-17T07:13:51.625 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-09-17T07:13:51.625 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-09-17T07:13:51.625 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-09-17T07:13:51.625 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-09-17T07:13:51.625 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-09-17T07:13:51.625 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-09-17T07:13:51.626 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-09-17T07:13:51.626 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-09-17T07:13:51.626 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-09-17T07:13:51.626 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-09-17T07:13:51.626 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-09-17T07:13:51.626 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-09-17T07:13:51.627 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-09-17T07:13:51.627 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-09-17T07:13:51.627 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-09-17T07:13:51.627 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-09-17T07:13:51.627 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-09-17T07:13:51.627 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-09-17T07:13:51.627 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-09-17T07:13:51.628 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-09-17T07:13:51.628 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-09-17T07:13:51.628 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-09-17T07:13:51.628 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-09-17T07:13:51.628 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-09-17T07:13:51.628 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-09-17T07:13:51.629 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-09-17T07:13:51.629 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-09-17T07:13:51.629 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-09-17T07:13:51.629 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-09-17T07:13:51.629 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-09-17T07:13:51.629 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-09-17T07:13:51.630 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-09-17T07:13:51.630 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-17T07:13:51.630 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-17T07:13:51.630 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-09-17T07:13:51.630 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-09-17T07:13:51.630 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-09-17T07:13:51.631 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-09-17T07:13:51.631 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-17T07:13:51.631 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-09-17T07:13:51.631 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-09-17T07:13:51.631 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-17T07:13:51.631 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-09-17T07:13:51.631 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-09-17T07:13:51.632 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-09-17T07:13:51.632 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-09-17T07:13:51.632 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-09-17T07:13:51.632 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-09-17T07:13:51.632 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-09-17T07:13:51.632 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-09-17T07:13:51.632 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-09-17T07:13:51.633 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-09-17T07:13:51.633 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-09-17T07:13:51.633 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-09-17T07:13:51.633 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-09-17T07:13:51.633 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-09-17T07:13:51.633 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-09-17T07:13:51.633 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-09-17T07:13:51.634 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-09-17T07:13:51.634 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-09-17T07:13:51.634 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-09-17T07:13:51.634 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-09-17T07:13:51.634 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-17T07:13:51.634 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-09-17T07:13:51.635 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-09-17T07:13:51.635 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-09-17T07:13:51.635 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-09-17T07:13:51.635 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-09-17T07:13:51.635 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-09-17T07:13:51.636 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-09-17T07:13:51.636 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-09-17T07:13:51.636 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-09-17T07:13:51.636 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-17T07:13:51.636 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-09-17T07:13:51.636 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-09-17T07:13:51.637 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-17T07:13:51.637 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-09-17T07:13:51.637 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-09-17T07:13:51.637 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-09-17T07:13:51.637 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-09-17T07:13:51.637 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-09-17T07:13:51.638 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-09-17T07:13:51.638 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-09-17T07:13:51.638 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-09-17T07:13:51.638 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-09-17T07:13:51.638 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-09-17T07:13:51.638 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-09-17T07:13:51.638 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-09-17T07:13:51.638 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-09-17T07:13:51.639 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-09-17T07:13:51.639 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-09-17T07:13:51.639 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-09-17T07:13:51.639 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-09-17T07:13:51.639 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-17T07:13:51.639 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-09-17T07:13:51.639 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-09-17T07:13:51.639 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-09-17T07:13:51.639 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-09-17T07:13:51.640 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-09-17T07:13:51.640 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-09-17T07:13:51.640 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-09-17T07:13:51.640 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-09-17T07:13:51.640 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-09-17T07:13:51.640 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-09-17T07:13:51.640 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-09-17T07:13:51.641 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-17T07:13:51.641 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-17T07:13:51.641 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-17T07:13:51.641 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-17T07:13:51.641 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-09-17T07:13:51.641 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-09-17T07:13:51.641 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-09-17T07:13:51.642 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-09-17T07:13:51.642 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-17T07:13:51.642 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-09-17T07:13:51.642 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-09-17T07:13:51.642 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-09-17T07:13:51.642 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-09-17T07:13:51.643 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-09-17T07:13:51.643 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-09-17T07:13:51.643 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-09-17T07:13:51.643 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-09-17T07:13:51.643 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-09-17T07:13:51.643 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-09-17T07:13:51.643 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-09-17T07:13:51.644 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-09-17T07:13:51.644 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-09-17T07:13:51.644 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-09-17T07:13:51.644 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-17T07:13:51.644 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-17T07:13:51.644 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-09-17T07:13:51.645 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-09-17T07:13:51.645 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-09-17T07:13:51.645 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-09-17T07:13:51.645 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-09-17T07:13:51.645 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-09-17T07:13:51.645 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-17T07:13:51.646 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-09-17T07:13:51.646 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-09-17T07:13:51.646 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 bash[36538]: 2436fcd841b8e31657c32d9448424d95e0f4c57548d9c3f03bef30d53eb002f6 2024-09-17T07:13:51.646 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-09-17T07:13:51.646 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-09-17T07:13:51.646 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-09-17T07:13:51.647 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-17T07:13:51.648 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-09-17T07:13:51.648 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-09-17T07:13:51.648 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-09-17T07:13:51.648 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 systemd[1]: Started Ceph grafana.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:13:51.648 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-09-17T07:13:51.648 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-09-17T07:13:51.648 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-09-17T07:13:51.648 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-09-17T07:13:51.648 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-09-17T07:13:51.648 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-09-17T07:13:51.649 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-09-17T07:13:51.649 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-09-17T07:13:51.649 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-09-17T07:13:51.649 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-09-17T07:13:51.649 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Created default admin" logger=sqlstore user=admin 2024-09-17T07:13:51.649 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Initializing HTTPServer" logger=server 2024-09-17T07:13:51.649 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Initializing BackendPluginManager" logger=server 2024-09-17T07:13:51.649 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Initializing PluginManager" logger=server 2024-09-17T07:13:51.649 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Starting plugin search" logger=plugins 2024-09-17T07:13:51.649 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Registering plugin" logger=plugins name="Pie Chart" 2024-09-17T07:13:51.649 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Registering plugin" logger=plugins name="Status Panel" 2024-09-17T07:13:51.649 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Initializing HooksService" logger=server 2024-09-17T07:13:51.650 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Initializing OSSLicensingService" logger=server 2024-09-17T07:13:51.650 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Initializing InternalMetricsService" logger=server 2024-09-17T07:13:51.650 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Initializing RemoteCache" logger=server 2024-09-17T07:13:51.650 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Initializing RenderingService" logger=server 2024-09-17T07:13:51.650 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Initializing AlertEngine" logger=server 2024-09-17T07:13:51.650 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Initializing QuotaService" logger=server 2024-09-17T07:13:51.650 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Initializing ServerLockService" logger=server 2024-09-17T07:13:51.650 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Initializing UserAuthTokenService" logger=server 2024-09-17T07:13:51.650 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Initializing DatasourceCacheService" logger=server 2024-09-17T07:13:51.650 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Initializing LoginService" logger=server 2024-09-17T07:13:51.650 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Initializing SearchService" logger=server 2024-09-17T07:13:51.650 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Initializing TracingService" logger=server 2024-09-17T07:13:51.650 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Initializing UsageStatsService" logger=server 2024-09-17T07:13:51.650 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Initializing CleanUpService" logger=server 2024-09-17T07:13:51.651 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Initializing NotificationService" logger=server 2024-09-17T07:13:51.651 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Initializing provisioningServiceImpl" logger=server 2024-09-17T07:13:51.651 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-09-17T07:13:51.651 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 2024-09-17T07:13:51.651 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Backend rendering via phantomJS" logger=rendering renderer=phantomJS 2024-09-17T07:13:51.651 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=warn msg="phantomJS is deprecated and will be removed in a future release. You should consider migrating from phantomJS to grafana-image-renderer plugin. Read more at https://grafana.com/docs/grafana/latest/administration/image_rendering/" logger=rendering renderer=phantomJS 2024-09-17T07:13:51.651 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="Initializing Stream Manager" 2024-09-17T07:13:51.651 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:13:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:13:51+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-09-17T07:13:52.522 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:50.866409+0000 mon.a (mon.0) 495 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-09-17T07:13:52.522 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:50.866842+0000 mon.a (mon.0) 496 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2024-09-17T07:13:52.522 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:51.312201+0000 mon.a (mon.0) 497 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:13:52.522 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:51.313277+0000 mon.a (mon.0) 498 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:13:52.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:50.866409+0000 mon.a (mon.0) 495 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-09-17T07:13:52.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:50.866842+0000 mon.a (mon.0) 496 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2024-09-17T07:13:52.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:51.312201+0000 mon.a (mon.0) 497 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:13:52.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:51.313277+0000 mon.a (mon.0) 498 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:13:52.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:50.866409+0000 mon.a (mon.0) 495 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-09-17T07:13:52.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:50.866842+0000 mon.a (mon.0) 496 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2024-09-17T07:13:52.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:51.312201+0000 mon.a (mon.0) 497 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:13:52.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:51.313277+0000 mon.a (mon.0) 498 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:13:53.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:51.783376+0000 mgr.y (mgr.14144) 213 : cluster [DBG] pgmap v199: 97 pgs: 7 creating+activating, 6 creating+peering, 21 unknown, 63 active+clean; 3.6 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.0 KiB/s wr, 14 op/s; 72 B/s, 0 objects/s recovering 2024-09-17T07:13:53.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:52.120033+0000 mon.a (mon.0) 499 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2024-09-17T07:13:53.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:52.127597+0000 mon.a (mon.0) 500 : audit [INF] from='client.? 172.21.15.44:0/3109628867' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T07:13:53.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:53.120874+0000 mon.a (mon.0) 501 : audit [INF] from='client.? 172.21.15.44:0/3109628867' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-09-17T07:13:53.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:53.120994+0000 mon.a (mon.0) 502 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2024-09-17T07:13:53.542 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:53 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:51.783376+0000 mgr.y (mgr.14144) 213 : cluster [DBG] pgmap v199: 97 pgs: 7 creating+activating, 6 creating+peering, 21 unknown, 63 active+clean; 3.6 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.0 KiB/s wr, 14 op/s; 72 B/s, 0 objects/s recovering 2024-09-17T07:13:53.542 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:53 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:52.120033+0000 mon.a (mon.0) 499 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2024-09-17T07:13:53.542 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:53 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:52.127597+0000 mon.a (mon.0) 500 : audit [INF] from='client.? 172.21.15.44:0/3109628867' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T07:13:53.542 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:53 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:53.120874+0000 mon.a (mon.0) 501 : audit [INF] from='client.? 172.21.15.44:0/3109628867' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-09-17T07:13:53.542 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:53 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:53.120994+0000 mon.a (mon.0) 502 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2024-09-17T07:13:53.542 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:53 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:51.783376+0000 mgr.y (mgr.14144) 213 : cluster [DBG] pgmap v199: 97 pgs: 7 creating+activating, 6 creating+peering, 21 unknown, 63 active+clean; 3.6 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.0 KiB/s wr, 14 op/s; 72 B/s, 0 objects/s recovering 2024-09-17T07:13:53.542 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:53 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:52.120033+0000 mon.a (mon.0) 499 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2024-09-17T07:13:53.543 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:53 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:52.127597+0000 mon.a (mon.0) 500 : audit [INF] from='client.? 172.21.15.44:0/3109628867' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T07:13:53.543 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:53 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:53.120874+0000 mon.a (mon.0) 501 : audit [INF] from='client.? 172.21.15.44:0/3109628867' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-09-17T07:13:53.543 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:53 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:53.120994+0000 mon.a (mon.0) 502 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2024-09-17T07:13:54.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:53.126895+0000 mon.a (mon.0) 503 : audit [INF] from='client.? 172.21.15.44:0/3109628867' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T07:13:54.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:54.018980+0000 mon.a (mon.0) 504 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:13:54.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:54.128888+0000 mon.a (mon.0) 505 : audit [INF] from='client.? 172.21.15.44:0/3109628867' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-17T07:13:54.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:54.128977+0000 mon.a (mon.0) 506 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2024-09-17T07:13:54.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:54.129554+0000 mon.a (mon.0) 507 : audit [INF] from='client.? 172.21.15.44:0/3109628867' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-09-17T07:13:54.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:53.126895+0000 mon.a (mon.0) 503 : audit [INF] from='client.? 172.21.15.44:0/3109628867' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T07:13:54.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:54.018980+0000 mon.a (mon.0) 504 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:13:54.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:54.128888+0000 mon.a (mon.0) 505 : audit [INF] from='client.? 172.21.15.44:0/3109628867' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-17T07:13:54.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:54.128977+0000 mon.a (mon.0) 506 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2024-09-17T07:13:54.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:54.129554+0000 mon.a (mon.0) 507 : audit [INF] from='client.? 172.21.15.44:0/3109628867' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-09-17T07:13:54.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:53.126895+0000 mon.a (mon.0) 503 : audit [INF] from='client.? 172.21.15.44:0/3109628867' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T07:13:54.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:54.018980+0000 mon.a (mon.0) 504 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:13:54.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:54.128888+0000 mon.a (mon.0) 505 : audit [INF] from='client.? 172.21.15.44:0/3109628867' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-17T07:13:54.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:54.128977+0000 mon.a (mon.0) 506 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2024-09-17T07:13:54.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:54.129554+0000 mon.a (mon.0) 507 : audit [INF] from='client.? 172.21.15.44:0/3109628867' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-09-17T07:13:55.218 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:13:55.218 INFO:teuthology.orchestra.run.smithi044.stdout: "id": "808b370f-632e-47fc-9b63-c2ff4e96a9b8", 2024-09-17T07:13:55.218 INFO:teuthology.orchestra.run.smithi044.stdout: "epoch": 1, 2024-09-17T07:13:55.218 INFO:teuthology.orchestra.run.smithi044.stdout: "predecessor_uuid": "00b9c429-3f2a-474a-8f09-956d28bc0a36", 2024-09-17T07:13:55.218 INFO:teuthology.orchestra.run.smithi044.stdout: "sync_status": [], 2024-09-17T07:13:55.218 INFO:teuthology.orchestra.run.smithi044.stdout: "period_map": { 2024-09-17T07:13:55.218 INFO:teuthology.orchestra.run.smithi044.stdout: "id": "808b370f-632e-47fc-9b63-c2ff4e96a9b8", 2024-09-17T07:13:55.218 INFO:teuthology.orchestra.run.smithi044.stdout: "zonegroups": [ 2024-09-17T07:13:55.218 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-09-17T07:13:55.218 INFO:teuthology.orchestra.run.smithi044.stdout: "id": "2e99a19f-76a1-4791-939d-7bc9089c963b", 2024-09-17T07:13:55.218 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "default", 2024-09-17T07:13:55.218 INFO:teuthology.orchestra.run.smithi044.stdout: "api_name": "default", 2024-09-17T07:13:55.219 INFO:teuthology.orchestra.run.smithi044.stdout: "is_master": "true", 2024-09-17T07:13:55.219 INFO:teuthology.orchestra.run.smithi044.stdout: "endpoints": [], 2024-09-17T07:13:55.219 INFO:teuthology.orchestra.run.smithi044.stdout: "hostnames": [], 2024-09-17T07:13:55.219 INFO:teuthology.orchestra.run.smithi044.stdout: "hostnames_s3website": [], 2024-09-17T07:13:55.219 INFO:teuthology.orchestra.run.smithi044.stdout: "master_zone": "86dcd634-218d-4e94-8f3e-3feca791c62c", 2024-09-17T07:13:55.219 INFO:teuthology.orchestra.run.smithi044.stdout: "zones": [ 2024-09-17T07:13:55.219 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-09-17T07:13:55.219 INFO:teuthology.orchestra.run.smithi044.stdout: "id": "86dcd634-218d-4e94-8f3e-3feca791c62c", 2024-09-17T07:13:55.219 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "z", 2024-09-17T07:13:55.219 INFO:teuthology.orchestra.run.smithi044.stdout: "endpoints": [], 2024-09-17T07:13:55.219 INFO:teuthology.orchestra.run.smithi044.stdout: "log_meta": "false", 2024-09-17T07:13:55.219 INFO:teuthology.orchestra.run.smithi044.stdout: "log_data": "false", 2024-09-17T07:13:55.219 INFO:teuthology.orchestra.run.smithi044.stdout: "bucket_index_max_shards": 11, 2024-09-17T07:13:55.219 INFO:teuthology.orchestra.run.smithi044.stdout: "read_only": "false", 2024-09-17T07:13:55.220 INFO:teuthology.orchestra.run.smithi044.stdout: "tier_type": "", 2024-09-17T07:13:55.220 INFO:teuthology.orchestra.run.smithi044.stdout: "sync_from_all": "true", 2024-09-17T07:13:55.220 INFO:teuthology.orchestra.run.smithi044.stdout: "sync_from": [], 2024-09-17T07:13:55.220 INFO:teuthology.orchestra.run.smithi044.stdout: "redirect_zone": "" 2024-09-17T07:13:55.220 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-09-17T07:13:55.220 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-09-17T07:13:55.220 INFO:teuthology.orchestra.run.smithi044.stdout: "placement_targets": [ 2024-09-17T07:13:55.220 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-09-17T07:13:55.220 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "default-placement", 2024-09-17T07:13:55.220 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-09-17T07:13:55.220 INFO:teuthology.orchestra.run.smithi044.stdout: "storage_classes": [ 2024-09-17T07:13:55.220 INFO:teuthology.orchestra.run.smithi044.stdout: "STANDARD" 2024-09-17T07:13:55.220 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-09-17T07:13:55.220 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-09-17T07:13:55.221 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-09-17T07:13:55.221 INFO:teuthology.orchestra.run.smithi044.stdout: "default_placement": "default-placement", 2024-09-17T07:13:55.221 INFO:teuthology.orchestra.run.smithi044.stdout: "realm_id": "e4999de9-4607-4aa1-95cb-dde6f41e8c30", 2024-09-17T07:13:55.221 INFO:teuthology.orchestra.run.smithi044.stdout: "sync_policy": { 2024-09-17T07:13:55.221 INFO:teuthology.orchestra.run.smithi044.stdout: "groups": [] 2024-09-17T07:13:55.221 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-09-17T07:13:55.221 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-09-17T07:13:55.221 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-09-17T07:13:55.221 INFO:teuthology.orchestra.run.smithi044.stdout: "short_zone_ids": [ 2024-09-17T07:13:55.221 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-09-17T07:13:55.221 INFO:teuthology.orchestra.run.smithi044.stdout: "key": "86dcd634-218d-4e94-8f3e-3feca791c62c", 2024-09-17T07:13:55.221 INFO:teuthology.orchestra.run.smithi044.stdout: "val": 1984558831 2024-09-17T07:13:55.221 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-09-17T07:13:55.221 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-09-17T07:13:55.222 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:13:55.222 INFO:teuthology.orchestra.run.smithi044.stdout: "master_zonegroup": "2e99a19f-76a1-4791-939d-7bc9089c963b", 2024-09-17T07:13:55.222 INFO:teuthology.orchestra.run.smithi044.stdout: "master_zone": "86dcd634-218d-4e94-8f3e-3feca791c62c", 2024-09-17T07:13:55.222 INFO:teuthology.orchestra.run.smithi044.stdout: "period_config": { 2024-09-17T07:13:55.222 INFO:teuthology.orchestra.run.smithi044.stdout: "bucket_quota": { 2024-09-17T07:13:55.222 INFO:teuthology.orchestra.run.smithi044.stdout: "enabled": false, 2024-09-17T07:13:55.222 INFO:teuthology.orchestra.run.smithi044.stdout: "check_on_raw": false, 2024-09-17T07:13:55.222 INFO:teuthology.orchestra.run.smithi044.stdout: "max_size": -1, 2024-09-17T07:13:55.222 INFO:teuthology.orchestra.run.smithi044.stdout: "max_size_kb": 0, 2024-09-17T07:13:55.222 INFO:teuthology.orchestra.run.smithi044.stdout: "max_objects": -1 2024-09-17T07:13:55.222 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:13:55.222 INFO:teuthology.orchestra.run.smithi044.stdout: "user_quota": { 2024-09-17T07:13:55.222 INFO:teuthology.orchestra.run.smithi044.stdout: "enabled": false, 2024-09-17T07:13:55.222 INFO:teuthology.orchestra.run.smithi044.stdout: "check_on_raw": false, 2024-09-17T07:13:55.223 INFO:teuthology.orchestra.run.smithi044.stdout: "max_size": -1, 2024-09-17T07:13:55.223 INFO:teuthology.orchestra.run.smithi044.stdout: "max_size_kb": 0, 2024-09-17T07:13:55.223 INFO:teuthology.orchestra.run.smithi044.stdout: "max_objects": -1 2024-09-17T07:13:55.223 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-09-17T07:13:55.223 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:13:55.223 INFO:teuthology.orchestra.run.smithi044.stdout: "realm_id": "e4999de9-4607-4aa1-95cb-dde6f41e8c30", 2024-09-17T07:13:55.223 INFO:teuthology.orchestra.run.smithi044.stdout: "realm_name": "r", 2024-09-17T07:13:55.223 INFO:teuthology.orchestra.run.smithi044.stdout: "realm_epoch": 2 2024-09-17T07:13:55.223 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:13:55.534 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:53.783821+0000 mgr.y (mgr.14144) 214 : cluster [DBG] pgmap v202: 129 pgs: 3 creating+peering, 7 creating+activating, 32 unknown, 87 active+clean; 9.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 12 KiB/s wr, 51 op/s; 72 B/s, 0 objects/s recovering 2024-09-17T07:13:55.534 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:55.054372+0000 mon.a (mon.0) 508 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:13:55.534 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:55.126298+0000 mon.a (mon.0) 509 : audit [INF] from='client.? 172.21.15.44:0/3109628867' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-09-17T07:13:55.534 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:55.126390+0000 mon.a (mon.0) 510 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2024-09-17T07:13:55.535 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:53.783821+0000 mgr.y (mgr.14144) 214 : cluster [DBG] pgmap v202: 129 pgs: 3 creating+peering, 7 creating+activating, 32 unknown, 87 active+clean; 9.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 12 KiB/s wr, 51 op/s; 72 B/s, 0 objects/s recovering 2024-09-17T07:13:55.535 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:55.054372+0000 mon.a (mon.0) 508 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:13:55.535 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:55.126298+0000 mon.a (mon.0) 509 : audit [INF] from='client.? 172.21.15.44:0/3109628867' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-09-17T07:13:55.535 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:55.126390+0000 mon.a (mon.0) 510 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2024-09-17T07:13:55.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:53.783821+0000 mgr.y (mgr.14144) 214 : cluster [DBG] pgmap v202: 129 pgs: 3 creating+peering, 7 creating+activating, 32 unknown, 87 active+clean; 9.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 12 KiB/s wr, 51 op/s; 72 B/s, 0 objects/s recovering 2024-09-17T07:13:55.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:55.054372+0000 mon.a (mon.0) 508 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:13:55.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:55.126298+0000 mon.a (mon.0) 509 : audit [INF] from='client.? 172.21.15.44:0/3109628867' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-09-17T07:13:55.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:55.126390+0000 mon.a (mon.0) 510 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2024-09-17T07:13:56.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:13:55.275756+0000 mon.a (mon.0) 511 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:13:56.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:13:55.277834+0000 mgr.y (mgr.14144) 215 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-09-17T07:13:56.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:13:55.280145+0000 mgr.y (mgr.14144) 216 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi044 2024-09-17T07:13:56.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:13:55.275756+0000 mon.a (mon.0) 511 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:13:56.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:13:55.277834+0000 mgr.y (mgr.14144) 215 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-09-17T07:13:56.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:13:55.280145+0000 mgr.y (mgr.14144) 216 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi044 2024-09-17T07:13:56.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:56 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:13:55.275756+0000 mon.a (mon.0) 511 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:13:56.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:56 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:13:55.277834+0000 mgr.y (mgr.14144) 215 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-09-17T07:13:56.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:56 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:13:55.280145+0000 mgr.y (mgr.14144) 216 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi044 2024-09-17T07:13:56.793 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-09-17T07:13:57.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:55.784288+0000 mgr.y (mgr.14144) 217 : cluster [DBG] pgmap v205: 129 pgs: 3 creating+peering, 7 creating+activating, 22 unknown, 97 active+clean; 15 KiB data, 46 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 19 KiB/s wr, 76 op/s 2024-09-17T07:13:57.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:55.784288+0000 mgr.y (mgr.14144) 217 : cluster [DBG] pgmap v205: 129 pgs: 3 creating+peering, 7 creating+activating, 22 unknown, 97 active+clean; 15 KiB data, 46 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 19 KiB/s wr, 76 op/s 2024-09-17T07:13:57.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:55.784288+0000 mgr.y (mgr.14144) 217 : cluster [DBG] pgmap v205: 129 pgs: 3 creating+peering, 7 creating+activating, 22 unknown, 97 active+clean; 15 KiB data, 46 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 19 KiB/s wr, 76 op/s 2024-09-17T07:13:59.496 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:13:59 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:57.784879+0000 mgr.y (mgr.14144) 218 : cluster [DBG] pgmap v206: 129 pgs: 3 creating+peering, 126 active+clean; 26 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 24 KiB/s wr, 95 op/s 2024-09-17T07:13:59.497 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:13:59 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:57.784879+0000 mgr.y (mgr.14144) 218 : cluster [DBG] pgmap v206: 129 pgs: 3 creating+peering, 126 active+clean; 26 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 24 KiB/s wr, 95 op/s 2024-09-17T07:13:59.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:13:59 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:57.784879+0000 mgr.y (mgr.14144) 218 : cluster [DBG] pgmap v206: 129 pgs: 3 creating+peering, 126 active+clean; 26 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 24 KiB/s wr, 95 op/s 2024-09-17T07:14:00.519 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:14:00 smithi044 systemd[1]: Stopping Ceph alertmanager.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:14:00.733 INFO:teuthology.orchestra.run.smithi044.stdout:Scheduled rgw.foo update... 2024-09-17T07:14:00.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:13:59.785352+0000 mgr.y (mgr.14144) 219 : cluster [DBG] pgmap v207: 129 pgs: 129 active+clean; 26 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 17 KiB/s wr, 68 op/s 2024-09-17T07:14:00.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:13:59.785352+0000 mgr.y (mgr.14144) 219 : cluster [DBG] pgmap v207: 129 pgs: 129 active+clean; 26 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 17 KiB/s wr, 68 op/s 2024-09-17T07:14:00.825 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:14:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a[41732]: level=info ts=2024-09-17T07:14:00.656Z caller=main.go:536 msg="Received SIGTERM, exiting gracefully..." 2024-09-17T07:14:00.825 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:14:00 smithi044 podman[42964]: 2024-09-17 07:14:00.676793406 +0000 UTC m=+0.327292419 container died f43892f6f8a5af635fef1df0d7aa55f2253b4ec36c9d04392f017fa2a3596f02 (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a, maintainer=The Prometheus Authors ) 2024-09-17T07:14:01.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:00 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:13:59.785352+0000 mgr.y (mgr.14144) 219 : cluster [DBG] pgmap v207: 129 pgs: 129 active+clean; 26 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 17 KiB/s wr, 68 op/s 2024-09-17T07:14:01.324 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:14:01 smithi044 podman[42964]: 2024-09-17 07:14:01.156851221 +0000 UTC m=+0.807350234 container cleanup f43892f6f8a5af635fef1df0d7aa55f2253b4ec36c9d04392f017fa2a3596f02 (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a, maintainer=The Prometheus Authors ) 2024-09-17T07:14:01.324 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:14:01 smithi044 podman[42964]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a 2024-09-17T07:14:01.788 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'ceph orch apply rgw smpl' 2024-09-17T07:14:02.036 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:00.728025+0000 mgr.y (mgr.14144) 220 : audit [DBG] from='client.24547 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:14:02.037 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:14:00.728927+0000 mgr.y (mgr.14144) 221 : cephadm [INF] Saving service rgw.foo spec with placement count:2 2024-09-17T07:14:02.037 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:00.732550+0000 mon.a (mon.0) 512 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:02.037 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:00.728025+0000 mgr.y (mgr.14144) 220 : audit [DBG] from='client.24547 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:14:02.037 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:14:00.728927+0000 mgr.y (mgr.14144) 221 : cephadm [INF] Saving service rgw.foo spec with placement count:2 2024-09-17T07:14:02.037 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:00.732550+0000 mon.a (mon.0) 512 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:02.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:01 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:00.728025+0000 mgr.y (mgr.14144) 220 : audit [DBG] from='client.24547 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:14:02.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:01 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:14:00.728927+0000 mgr.y (mgr.14144) 221 : cephadm [INF] Saving service rgw.foo spec with placement count:2 2024-09-17T07:14:02.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:01 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:00.732550+0000 mon.a (mon.0) 512 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:02.325 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:14:02 smithi044 podman[42998]: 2024-09-17 07:14:02.036374004 +0000 UTC m=+1.358530066 container remove f43892f6f8a5af635fef1df0d7aa55f2253b4ec36c9d04392f017fa2a3596f02 (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a, maintainer=The Prometheus Authors ) 2024-09-17T07:14:02.325 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:14:02 smithi044 podman[42998]: 2024-09-17 07:14:02.137334252 +0000 UTC m=+1.459490313 volume remove 3fbf3caceb635d2d9b9e7f10229d5e0b581e13828c5dc3fcb974004172ee87af 2024-09-17T07:14:02.325 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:14:02 smithi044 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-17T07:14:02.325 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:14:02 smithi044 systemd[1]: Stopped Ceph alertmanager.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:14:02.325 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:14:02 smithi044 systemd[1]: Starting Ceph alertmanager.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:14:03.054 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:01.786131+0000 mgr.y (mgr.14144) 222 : cluster [DBG] pgmap v208: 129 pgs: 129 active+clean; 27 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 15 KiB/s wr, 57 op/s 2024-09-17T07:14:03.055 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:01.786131+0000 mgr.y (mgr.14144) 222 : cluster [DBG] pgmap v208: 129 pgs: 129 active+clean; 27 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 15 KiB/s wr, 57 op/s 2024-09-17T07:14:03.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:01.786131+0000 mgr.y (mgr.14144) 222 : cluster [DBG] pgmap v208: 129 pgs: 129 active+clean; 27 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 15 KiB/s wr, 57 op/s 2024-09-17T07:14:03.324 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:14:03 smithi044 podman[43119]: 2024-09-17 07:14:02.957850151 +0000 UTC m=+0.019903519 image pull 0881eb8f169f5556a292b4e2c01d683172b12830a62a9225a98a8e206bb734f0 docker.io/prom/alertmanager:v0.20.0 2024-09-17T07:14:03.976 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:14:03 smithi044 podman[43119]: 2024-09-17 07:14:03.781675393 +0000 UTC m=+0.843728745 volume create 1ecb3ab103e7da8548d62f0456d8038503692201d44ada00a39fba39cc226e77 2024-09-17T07:14:03.976 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:14:03 smithi044 podman[43119]: 2024-09-17 07:14:03.873831329 +0000 UTC m=+0.935884724 container create 42aa02852f042d4d5857f976c8b3c45e5a5949b833fd1dc725707c52e9f978ca (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a, maintainer=The Prometheus Authors ) 2024-09-17T07:14:04.300 INFO:teuthology.orchestra.run.smithi044.stdout:Scheduled rgw.smpl update... 2024-09-17T07:14:04.575 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:14:04 smithi044 podman[43119]: 2024-09-17 07:14:04.352417242 +0000 UTC m=+1.414470601 container init 42aa02852f042d4d5857f976c8b3c45e5a5949b833fd1dc725707c52e9f978ca (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a, maintainer=The Prometheus Authors ) 2024-09-17T07:14:04.575 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:14:04 smithi044 podman[43119]: 2024-09-17 07:14:04.355436726 +0000 UTC m=+1.417490076 container start 42aa02852f042d4d5857f976c8b3c45e5a5949b833fd1dc725707c52e9f978ca (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a, maintainer=The Prometheus Authors ) 2024-09-17T07:14:04.575 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:14:04 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a[43179]: level=info ts=2024-09-17T07:14:04.375Z caller=main.go:231 msg="Starting Alertmanager" version="(version=0.20.0, branch=HEAD, revision=f74be0400a6243d10bb53812d6fa408ad71ff32d)" 2024-09-17T07:14:04.575 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:14:04 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a[43179]: level=info ts=2024-09-17T07:14:04.375Z caller=main.go:232 build_context="(go=go1.13.5, user=root@00c3106655f8, date=20191211-14:13:14)" 2024-09-17T07:14:04.575 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:14:04 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a[43179]: level=info ts=2024-09-17T07:14:04.376Z caller=cluster.go:161 component=cluster msg="setting advertise address explicitly" addr=172.21.15.44 port=9094 2024-09-17T07:14:04.575 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:14:04 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a[43179]: level=info ts=2024-09-17T07:14:04.377Z caller=cluster.go:623 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-17T07:14:04.575 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:14:04 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a[43179]: level=info ts=2024-09-17T07:14:04.396Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-17T07:14:04.575 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:14:04 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a[43179]: level=info ts=2024-09-17T07:14:04.396Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-17T07:14:04.575 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:14:04 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a[43179]: level=info ts=2024-09-17T07:14:04.399Z caller=main.go:497 msg=Listening address=:9093 2024-09-17T07:14:04.576 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:14:04 smithi044 bash[43119]: 42aa02852f042d4d5857f976c8b3c45e5a5949b833fd1dc725707c52e9f978ca 2024-09-17T07:14:04.576 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:14:04 smithi044 systemd[1]: Started Ceph alertmanager.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:14:05.064 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'ceph osd pool create foo' 2024-09-17T07:14:05.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:05 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:03.786747+0000 mgr.y (mgr.14144) 223 : cluster [DBG] pgmap v209: 129 pgs: 129 active+clean; 27 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 8.9 KiB/s rd, 12 KiB/s wr, 47 op/s 2024-09-17T07:14:05.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:05 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:04.295444+0000 mgr.y (mgr.14144) 224 : audit [DBG] from='client.14634 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:14:05.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:05 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:14:04.295879+0000 mgr.y (mgr.14144) 225 : cephadm [INF] Saving service rgw.smpl spec with placement count:2 2024-09-17T07:14:05.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:05 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:04.299025+0000 mon.a (mon.0) 513 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:05.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:05 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:04.483107+0000 mon.a (mon.0) 514 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:05.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:05 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:14:04.483866+0000 mgr.y (mgr.14144) 226 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T07:14:05.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:05 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:14:04.485955+0000 mgr.y (mgr.14144) 227 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi179 2024-09-17T07:14:05.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:03.786747+0000 mgr.y (mgr.14144) 223 : cluster [DBG] pgmap v209: 129 pgs: 129 active+clean; 27 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 8.9 KiB/s rd, 12 KiB/s wr, 47 op/s 2024-09-17T07:14:05.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:04.295444+0000 mgr.y (mgr.14144) 224 : audit [DBG] from='client.14634 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:14:05.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:14:04.295879+0000 mgr.y (mgr.14144) 225 : cephadm [INF] Saving service rgw.smpl spec with placement count:2 2024-09-17T07:14:05.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:04.299025+0000 mon.a (mon.0) 513 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:05.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:04.483107+0000 mon.a (mon.0) 514 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:05.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:14:04.483866+0000 mgr.y (mgr.14144) 226 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T07:14:05.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:14:04.485955+0000 mgr.y (mgr.14144) 227 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi179 2024-09-17T07:14:05.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:03.786747+0000 mgr.y (mgr.14144) 223 : cluster [DBG] pgmap v209: 129 pgs: 129 active+clean; 27 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 8.9 KiB/s rd, 12 KiB/s wr, 47 op/s 2024-09-17T07:14:05.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:04.295444+0000 mgr.y (mgr.14144) 224 : audit [DBG] from='client.14634 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:14:05.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:14:04.295879+0000 mgr.y (mgr.14144) 225 : cephadm [INF] Saving service rgw.smpl spec with placement count:2 2024-09-17T07:14:05.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:04.299025+0000 mon.a (mon.0) 513 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:05.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:04.483107+0000 mon.a (mon.0) 514 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:05.578 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:14:04.483866+0000 mgr.y (mgr.14144) 226 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T07:14:05.578 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:14:04.485955+0000 mgr.y (mgr.14144) 227 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi179 2024-09-17T07:14:06.699 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:14:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a[43179]: level=info ts=2024-09-17T07:14:06.377Z caller=cluster.go:648 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000074009s 2024-09-17T07:14:07.586 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'foo' created 2024-09-17T07:14:07.689 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:05.787347+0000 mgr.y (mgr.14144) 228 : cluster [DBG] pgmap v210: 129 pgs: 129 active+clean; 27 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 6.9 KiB/s rd, 9.9 KiB/s wr, 38 op/s 2024-09-17T07:14:07.690 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:07 smithi179 systemd[1]: Stopping Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:14:07.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:05.787347+0000 mgr.y (mgr.14144) 228 : cluster [DBG] pgmap v210: 129 pgs: 129 active+clean; 27 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 6.9 KiB/s rd, 9.9 KiB/s wr, 38 op/s 2024-09-17T07:14:07.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:05.787347+0000 mgr.y (mgr.14144) 228 : cluster [DBG] pgmap v210: 129 pgs: 129 active+clean; 27 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 6.9 KiB/s rd, 9.9 KiB/s wr, 38 op/s 2024-09-17T07:14:07.966 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[35786]: level=warn ts=2024-09-17T07:14:07.689Z caller=main.go:524 msg="Received SIGTERM, exiting gracefully..." 2024-09-17T07:14:07.966 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[35786]: level=info ts=2024-09-17T07:14:07.689Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2024-09-17T07:14:07.966 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[35786]: level=info ts=2024-09-17T07:14:07.689Z caller=main.go:561 msg="Stopping notify discovery manager..." 2024-09-17T07:14:07.966 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[35786]: level=info ts=2024-09-17T07:14:07.689Z caller=main.go:583 msg="Stopping scrape manager..." 2024-09-17T07:14:07.966 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[35786]: level=info ts=2024-09-17T07:14:07.689Z caller=main.go:543 msg="Scrape discovery manager stopped" 2024-09-17T07:14:07.966 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[35786]: level=info ts=2024-09-17T07:14:07.689Z caller=main.go:557 msg="Notify discovery manager stopped" 2024-09-17T07:14:07.966 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[35786]: level=info ts=2024-09-17T07:14:07.690Z caller=main.go:577 msg="Scrape manager stopped" 2024-09-17T07:14:07.966 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[35786]: level=info ts=2024-09-17T07:14:07.690Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2024-09-17T07:14:07.966 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[35786]: level=info ts=2024-09-17T07:14:07.690Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2024-09-17T07:14:07.966 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[35786]: level=info ts=2024-09-17T07:14:07.690Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2024-09-17T07:14:07.966 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[35786]: level=info ts=2024-09-17T07:14:07.690Z caller=main.go:749 msg="Notifier manager stopped" 2024-09-17T07:14:07.967 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[35786]: level=info ts=2024-09-17T07:14:07.690Z caller=main.go:761 msg="See you next time!" 2024-09-17T07:14:07.967 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:07 smithi179 podman[37102]: 2024-09-17 07:14:07.700107895 +0000 UTC m=+0.134857037 container died 556d0546f78eea557b88ac45406fb8e874a1a446f267284df556444c036efdd3 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:14:08.274 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'rbd pool init foo' 2024-09-17T07:14:08.364 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:08 smithi179 podman[37102]: 2024-09-17 07:14:08.140004548 +0000 UTC m=+0.574753691 container cleanup 556d0546f78eea557b88ac45406fb8e874a1a446f267284df556444c036efdd3 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:14:08.364 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:08 smithi179 podman[37102]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a 2024-09-17T07:14:08.722 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:07.087944+0000 mon.a (mon.0) 515 : audit [INF] from='client.? 172.21.15.44:0/603272368' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-09-17T07:14:08.722 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:07.578552+0000 mon.a (mon.0) 516 : audit [INF] from='client.? 172.21.15.44:0/603272368' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-09-17T07:14:08.722 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:07.578618+0000 mon.a (mon.0) 517 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2024-09-17T07:14:08.722 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:08 smithi179 podman[37114]: 2024-09-17 07:14:08.400933996 +0000 UTC m=+0.701959933 container remove 556d0546f78eea557b88ac45406fb8e874a1a446f267284df556444c036efdd3 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:14:08.722 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:08 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-17T07:14:08.722 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:08 smithi179 systemd[1]: Stopped Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:14:08.722 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:08 smithi179 systemd[1]: Starting Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:14:08.722 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:08 smithi179 podman[37157]: 2024-09-17 07:14:08.496613634 +0000 UTC m=+0.017693648 image pull de242295e2257c37c8cadfd962369228f8f10b2d48a44259b65fef44ad4f6490 docker.io/prom/prometheus:v2.18.1 2024-09-17T07:14:08.722 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:08 smithi179 podman[37157]: 2024-09-17 07:14:08.609185402 +0000 UTC m=+0.130265438 container create 3b50713c05621649d5a83663e1a602c970e0f4bf1e11afbdbce3394dc595a10e (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:14:08.782 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:07.087944+0000 mon.a (mon.0) 515 : audit [INF] from='client.? 172.21.15.44:0/603272368' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-09-17T07:14:08.782 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:07.578552+0000 mon.a (mon.0) 516 : audit [INF] from='client.? 172.21.15.44:0/603272368' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-09-17T07:14:08.782 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:07.578618+0000 mon.a (mon.0) 517 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2024-09-17T07:14:08.782 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:07.087944+0000 mon.a (mon.0) 515 : audit [INF] from='client.? 172.21.15.44:0/603272368' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-09-17T07:14:08.782 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:07.578552+0000 mon.a (mon.0) 516 : audit [INF] from='client.? 172.21.15.44:0/603272368' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-09-17T07:14:08.782 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:07.578618+0000 mon.a (mon.0) 517 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2024-09-17T07:14:09.114 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:08 smithi179 podman[37157]: 2024-09-17 07:14:08.861407783 +0000 UTC m=+0.382487786 container init 3b50713c05621649d5a83663e1a602c970e0f4bf1e11afbdbce3394dc595a10e (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:14:09.114 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:08 smithi179 podman[37157]: 2024-09-17 07:14:08.864432403 +0000 UTC m=+0.385512400 container start 3b50713c05621649d5a83663e1a602c970e0f4bf1e11afbdbce3394dc595a10e (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:14:09.114 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[37172]: level=info ts=2024-09-17T07:14:08.895Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-17T07:14:09.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[37172]: level=info ts=2024-09-17T07:14:08.896Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2024-09-17T07:14:09.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[37172]: level=info ts=2024-09-17T07:14:08.896Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2024-09-17T07:14:09.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[37172]: level=info ts=2024-09-17T07:14:08.896Z caller=main.go:339 host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi179 (none))" 2024-09-17T07:14:09.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[37172]: level=info ts=2024-09-17T07:14:08.896Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T07:14:09.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[37172]: level=info ts=2024-09-17T07:14:08.896Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T07:14:09.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[37172]: level=info ts=2024-09-17T07:14:08.897Z caller=main.go:678 msg="Starting TSDB ..." 2024-09-17T07:14:09.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[37172]: level=info ts=2024-09-17T07:14:08.897Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2024-09-17T07:14:09.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[37172]: level=info ts=2024-09-17T07:14:08.900Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2024-09-17T07:14:09.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[37172]: level=info ts=2024-09-17T07:14:08.900Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-17T07:14:09.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[37172]: level=info ts=2024-09-17T07:14:08.900Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-17T07:14:09.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[37172]: level=info ts=2024-09-17T07:14:08.900Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=226.266µs 2024-09-17T07:14:09.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[37172]: level=info ts=2024-09-17T07:14:08.901Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2024-09-17T07:14:09.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[37172]: level=info ts=2024-09-17T07:14:08.901Z caller=main.go:695 msg="TSDB started" 2024-09-17T07:14:09.116 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[37172]: level=info ts=2024-09-17T07:14:08.901Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T07:14:09.116 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[37172]: level=info ts=2024-09-17T07:14:08.902Z caller=main.go:827 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T07:14:09.116 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[37172]: level=info ts=2024-09-17T07:14:08.902Z caller=main.go:646 msg="Server is ready to receive web requests." 2024-09-17T07:14:09.116 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:08 smithi179 bash[37157]: 3b50713c05621649d5a83663e1a602c970e0f4bf1e11afbdbce3394dc595a10e 2024-09-17T07:14:09.116 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:14:08 smithi179 systemd[1]: Started Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:14:09.771 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:07.787667+0000 mgr.y (mgr.14144) 229 : cluster [DBG] pgmap v212: 161 pgs: 32 unknown, 129 active+clean; 27 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 511 B/s wr, 1 op/s 2024-09-17T07:14:09.771 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:08.584629+0000 mon.a (mon.0) 518 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2024-09-17T07:14:09.771 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:08.980629+0000 mon.a (mon.0) 519 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:09.771 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:08.983067+0000 mon.a (mon.0) 520 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T07:14:09.771 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:08.984171+0000 mon.a (mon.0) 521 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi044:9093"}]: dispatch 2024-09-17T07:14:09.771 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:08.987828+0000 mon.a (mon.0) 522 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:09.772 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:08.991646+0000 mon.a (mon.0) 523 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T07:14:09.772 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:08.995870+0000 mon.a (mon.0) 524 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi179:9095"}]: dispatch 2024-09-17T07:14:09.772 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:08.999670+0000 mon.a (mon.0) 525 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:09.772 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:09.002564+0000 mon.a (mon.0) 526 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T07:14:09.772 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:09.006466+0000 mon.a (mon.0) 527 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi179:3000"}]: dispatch 2024-09-17T07:14:09.772 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:09.010636+0000 mon.a (mon.0) 528 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:09.772 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:09.012791+0000 mon.a (mon.0) 529 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:14:09.772 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:07.787667+0000 mgr.y (mgr.14144) 229 : cluster [DBG] pgmap v212: 161 pgs: 32 unknown, 129 active+clean; 27 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 511 B/s wr, 1 op/s 2024-09-17T07:14:09.773 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:08.584629+0000 mon.a (mon.0) 518 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2024-09-17T07:14:09.773 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:08.980629+0000 mon.a (mon.0) 519 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:09.773 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:08.983067+0000 mon.a (mon.0) 520 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T07:14:09.773 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:08.984171+0000 mon.a (mon.0) 521 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi044:9093"}]: dispatch 2024-09-17T07:14:09.773 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:08.987828+0000 mon.a (mon.0) 522 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:09.773 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:08.991646+0000 mon.a (mon.0) 523 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T07:14:09.773 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:08.995870+0000 mon.a (mon.0) 524 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi179:9095"}]: dispatch 2024-09-17T07:14:09.773 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:08.999670+0000 mon.a (mon.0) 525 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:09.773 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:09.002564+0000 mon.a (mon.0) 526 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T07:14:09.773 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:09.006466+0000 mon.a (mon.0) 527 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi179:3000"}]: dispatch 2024-09-17T07:14:09.773 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:09.010636+0000 mon.a (mon.0) 528 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:09.773 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:09.012791+0000 mon.a (mon.0) 529 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:14:09.774 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:07.787667+0000 mgr.y (mgr.14144) 229 : cluster [DBG] pgmap v212: 161 pgs: 32 unknown, 129 active+clean; 27 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 511 B/s wr, 1 op/s 2024-09-17T07:14:09.774 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:08.584629+0000 mon.a (mon.0) 518 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2024-09-17T07:14:09.774 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:08.980629+0000 mon.a (mon.0) 519 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:09.774 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:08.983067+0000 mon.a (mon.0) 520 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T07:14:09.774 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:08.984171+0000 mon.a (mon.0) 521 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi044:9093"}]: dispatch 2024-09-17T07:14:09.774 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:08.987828+0000 mon.a (mon.0) 522 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:09.774 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:08.991646+0000 mon.a (mon.0) 523 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T07:14:09.774 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:08.995870+0000 mon.a (mon.0) 524 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi179:9095"}]: dispatch 2024-09-17T07:14:09.774 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:08.999670+0000 mon.a (mon.0) 525 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:09.775 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:09.002564+0000 mon.a (mon.0) 526 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T07:14:09.775 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:09.006466+0000 mon.a (mon.0) 527 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi179:3000"}]: dispatch 2024-09-17T07:14:09.775 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:09.010636+0000 mon.a (mon.0) 528 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:09.775 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:09.012791+0000 mon.a (mon.0) 529 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:14:10.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:08.983361+0000 mgr.y (mgr.14144) 230 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T07:14:10.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:08.984402+0000 mgr.y (mgr.14144) 231 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi044:9093"}]: dispatch 2024-09-17T07:14:10.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:08.992129+0000 mgr.y (mgr.14144) 232 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T07:14:10.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:08.996164+0000 mgr.y (mgr.14144) 233 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi179:9095"}]: dispatch 2024-09-17T07:14:10.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:09.002884+0000 mgr.y (mgr.14144) 234 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T07:14:10.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:09.006735+0000 mgr.y (mgr.14144) 235 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi179:3000"}]: dispatch 2024-09-17T07:14:10.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:09.587632+0000 mon.a (mon.0) 530 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2024-09-17T07:14:10.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:09.953953+0000 mon.c (mon.1) 22 : audit [INF] from='client.? 172.21.15.44:0/569399906' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-09-17T07:14:10.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:09.954257+0000 mon.a (mon.0) 531 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-09-17T07:14:10.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:08.983361+0000 mgr.y (mgr.14144) 230 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T07:14:10.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:08.984402+0000 mgr.y (mgr.14144) 231 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi044:9093"}]: dispatch 2024-09-17T07:14:10.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:08.992129+0000 mgr.y (mgr.14144) 232 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T07:14:10.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:08.996164+0000 mgr.y (mgr.14144) 233 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi179:9095"}]: dispatch 2024-09-17T07:14:10.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:09.002884+0000 mgr.y (mgr.14144) 234 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T07:14:10.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:09.006735+0000 mgr.y (mgr.14144) 235 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi179:3000"}]: dispatch 2024-09-17T07:14:10.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:09.587632+0000 mon.a (mon.0) 530 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2024-09-17T07:14:10.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:09.953953+0000 mon.c (mon.1) 22 : audit [INF] from='client.? 172.21.15.44:0/569399906' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-09-17T07:14:10.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:09.954257+0000 mon.a (mon.0) 531 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-09-17T07:14:10.729 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:08.983361+0000 mgr.y (mgr.14144) 230 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T07:14:10.730 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:08.984402+0000 mgr.y (mgr.14144) 231 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi044:9093"}]: dispatch 2024-09-17T07:14:10.730 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:08.992129+0000 mgr.y (mgr.14144) 232 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T07:14:10.730 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:08.996164+0000 mgr.y (mgr.14144) 233 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi179:9095"}]: dispatch 2024-09-17T07:14:10.730 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:09.002884+0000 mgr.y (mgr.14144) 234 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T07:14:10.730 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:09.006735+0000 mgr.y (mgr.14144) 235 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi179:3000"}]: dispatch 2024-09-17T07:14:10.730 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:09.587632+0000 mon.a (mon.0) 530 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2024-09-17T07:14:10.730 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:09.953953+0000 mon.c (mon.1) 22 : audit [INF] from='client.? 172.21.15.44:0/569399906' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-09-17T07:14:10.730 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:09.954257+0000 mon.a (mon.0) 531 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-09-17T07:14:11.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:09.788114+0000 mgr.y (mgr.14144) 236 : cluster [DBG] pgmap v215: 161 pgs: 5 creating+peering, 27 unknown, 129 active+clean; 27 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:14:11.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:10.650883+0000 mon.a (mon.0) 532 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-09-17T07:14:11.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:10.650969+0000 mon.a (mon.0) 533 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2024-09-17T07:14:11.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:09.788114+0000 mgr.y (mgr.14144) 236 : cluster [DBG] pgmap v215: 161 pgs: 5 creating+peering, 27 unknown, 129 active+clean; 27 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:14:11.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:10.650883+0000 mon.a (mon.0) 532 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-09-17T07:14:11.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:10.650969+0000 mon.a (mon.0) 533 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2024-09-17T07:14:11.831 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:11 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:09.788114+0000 mgr.y (mgr.14144) 236 : cluster [DBG] pgmap v215: 161 pgs: 5 creating+peering, 27 unknown, 129 active+clean; 27 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:14:11.831 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:11 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:10.650883+0000 mon.a (mon.0) 532 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-09-17T07:14:11.831 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:11 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:10.650969+0000 mon.a (mon.0) 533 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2024-09-17T07:14:13.028 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:11.653496+0000 mon.a (mon.0) 534 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2024-09-17T07:14:13.028 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:11.788759+0000 mgr.y (mgr.14144) 237 : cluster [DBG] pgmap v218: 161 pgs: 5 creating+peering, 5 unknown, 151 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:14:13.028 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:12.325837+0000 mon.a (mon.0) 535 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:13.028 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:12.409088+0000 mon.a (mon.0) 536 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:13.028 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:12.414678+0000 mon.a (mon.0) 537 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:13.029 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:12.420314+0000 mon.a (mon.0) 538 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:13.029 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:12.424383+0000 mon.a (mon.0) 539 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:13.029 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:14:12.424898+0000 mgr.y (mgr.14144) 238 : cephadm [INF] Saving service rgw.foo spec with placement count:2 2024-09-17T07:14:13.029 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:12.427946+0000 mon.a (mon.0) 540 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:13.029 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:12.428527+0000 mon.a (mon.0) 541 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi044.nhyhnh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:14:13.029 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:12.431806+0000 mon.a (mon.0) 542 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi044.nhyhnh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T07:14:13.029 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:12.435981+0000 mon.a (mon.0) 543 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:13.029 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:12.436605+0000 mon.a (mon.0) 544 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:14:13.029 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:14:12.437161+0000 mgr.y (mgr.14144) 239 : cephadm [INF] Deploying daemon rgw.foo.smithi044.nhyhnh on smithi044 2024-09-17T07:14:13.029 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:11.653496+0000 mon.a (mon.0) 534 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2024-09-17T07:14:13.030 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:11.788759+0000 mgr.y (mgr.14144) 237 : cluster [DBG] pgmap v218: 161 pgs: 5 creating+peering, 5 unknown, 151 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:14:13.030 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:12.325837+0000 mon.a (mon.0) 535 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:13.030 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:12.409088+0000 mon.a (mon.0) 536 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:13.030 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:12.414678+0000 mon.a (mon.0) 537 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:13.030 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:12.420314+0000 mon.a (mon.0) 538 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:13.030 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:12.424383+0000 mon.a (mon.0) 539 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:13.030 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:14:12.424898+0000 mgr.y (mgr.14144) 238 : cephadm [INF] Saving service rgw.foo spec with placement count:2 2024-09-17T07:14:13.030 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:12.427946+0000 mon.a (mon.0) 540 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:13.030 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:12.428527+0000 mon.a (mon.0) 541 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi044.nhyhnh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:14:13.030 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:12.431806+0000 mon.a (mon.0) 542 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi044.nhyhnh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T07:14:13.030 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:12.435981+0000 mon.a (mon.0) 543 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:13.030 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:12.436605+0000 mon.a (mon.0) 544 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:14:13.030 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:14:12.437161+0000 mgr.y (mgr.14144) 239 : cephadm [INF] Deploying daemon rgw.foo.smithi044.nhyhnh on smithi044 2024-09-17T07:14:13.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:12 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:11.653496+0000 mon.a (mon.0) 534 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2024-09-17T07:14:13.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:12 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:11.788759+0000 mgr.y (mgr.14144) 237 : cluster [DBG] pgmap v218: 161 pgs: 5 creating+peering, 5 unknown, 151 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:14:13.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:12 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:12.325837+0000 mon.a (mon.0) 535 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:13.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:12 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:12.409088+0000 mon.a (mon.0) 536 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:13.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:12 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:12.414678+0000 mon.a (mon.0) 537 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:13.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:12 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:12.420314+0000 mon.a (mon.0) 538 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:13.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:12 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:12.424383+0000 mon.a (mon.0) 539 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:13.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:12 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:14:12.424898+0000 mgr.y (mgr.14144) 238 : cephadm [INF] Saving service rgw.foo spec with placement count:2 2024-09-17T07:14:13.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:12 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:12.427946+0000 mon.a (mon.0) 540 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:13.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:12 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:12.428527+0000 mon.a (mon.0) 541 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi044.nhyhnh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:14:13.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:12 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:12.431806+0000 mon.a (mon.0) 542 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi044.nhyhnh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T07:14:13.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:12 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:12.435981+0000 mon.a (mon.0) 543 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:13.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:12 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:12.436605+0000 mon.a (mon.0) 544 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:14:13.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:12 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:14:12.437161+0000 mgr.y (mgr.14144) 239 : cephadm [INF] Deploying daemon rgw.foo.smithi044.nhyhnh on smithi044 2024-09-17T07:14:13.967 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:12.659514+0000 mon.a (mon.0) 545 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2024-09-17T07:14:13.967 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:12.659514+0000 mon.a (mon.0) 545 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2024-09-17T07:14:14.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:13 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:12.659514+0000 mon.a (mon.0) 545 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2024-09-17T07:14:14.582 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:14:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a[43179]: level=info ts=2024-09-17T07:14:14.378Z caller=cluster.go:640 component=cluster msg="gossip settled; proceeding" elapsed=10.000600781s 2024-09-17T07:14:14.605 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'ceph orch apply iscsi foo u p' 2024-09-17T07:14:14.926 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:13.789287+0000 mgr.y (mgr.14144) 240 : cluster [DBG] pgmap v220: 161 pgs: 161 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 243 B/s wr, 0 op/s 2024-09-17T07:14:14.927 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:13.789287+0000 mgr.y (mgr.14144) 240 : cluster [DBG] pgmap v220: 161 pgs: 161 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 243 B/s wr, 0 op/s 2024-09-17T07:14:15.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:14 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:13.789287+0000 mgr.y (mgr.14144) 240 : cluster [DBG] pgmap v220: 161 pgs: 161 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 243 B/s wr, 0 op/s 2024-09-17T07:14:16.440 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:15.418932+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:16.440 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:15.419848+0000 mon.a (mon.0) 547 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi179.tqlzyy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:14:16.440 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:15.423263+0000 mon.a (mon.0) 548 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi179.tqlzyy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T07:14:16.440 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:15.429878+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:16.440 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:15.430594+0000 mon.a (mon.0) 550 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:14:16.441 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:14:15.431192+0000 mgr.y (mgr.14144) 241 : cephadm [INF] Deploying daemon rgw.foo.smithi179.tqlzyy on smithi179 2024-09-17T07:14:16.441 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:15.418932+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:16.441 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:15.419848+0000 mon.a (mon.0) 547 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi179.tqlzyy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:14:16.441 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:15.423263+0000 mon.a (mon.0) 548 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi179.tqlzyy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T07:14:16.441 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:15.429878+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:16.441 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:15.430594+0000 mon.a (mon.0) 550 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:14:16.441 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:14:15.431192+0000 mgr.y (mgr.14144) 241 : cephadm [INF] Deploying daemon rgw.foo.smithi179.tqlzyy on smithi179 2024-09-17T07:14:16.800 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:16 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:15.418932+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:16.800 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:16 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:15.419848+0000 mon.a (mon.0) 547 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi179.tqlzyy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:14:16.801 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:16 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:15.423263+0000 mon.a (mon.0) 548 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi179.tqlzyy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T07:14:16.801 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:16 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:15.429878+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:16.801 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:16 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:15.430594+0000 mon.a (mon.0) 550 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:14:16.801 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:16 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:14:15.431192+0000 mgr.y (mgr.14144) 241 : cephadm [INF] Deploying daemon rgw.foo.smithi179.tqlzyy on smithi179 2024-09-17T07:14:17.048 INFO:teuthology.orchestra.run.smithi044.stdout:Scheduled iscsi.foo update... 2024-09-17T07:14:17.640 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:17 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:15.789750+0000 mgr.y (mgr.14144) 242 : cluster [DBG] pgmap v221: 161 pgs: 161 active+clean; 27 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 7.7 KiB/s rd, 170 B/s wr, 14 op/s 2024-09-17T07:14:17.640 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:17 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:17.046740+0000 mon.a (mon.0) 551 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:17.640 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:17 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:15.789750+0000 mgr.y (mgr.14144) 242 : cluster [DBG] pgmap v221: 161 pgs: 161 active+clean; 27 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 7.7 KiB/s rd, 170 B/s wr, 14 op/s 2024-09-17T07:14:17.640 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:17 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:17.046740+0000 mon.a (mon.0) 551 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:17.663 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'sleep 120' 2024-09-17T07:14:17.792 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:15.789750+0000 mgr.y (mgr.14144) 242 : cluster [DBG] pgmap v221: 161 pgs: 161 active+clean; 27 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 7.7 KiB/s rd, 170 B/s wr, 14 op/s 2024-09-17T07:14:17.793 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:17.046740+0000 mon.a (mon.0) 551 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:18.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:17.043006+0000 mgr.y (mgr.14144) 243 : audit [DBG] from='client.24592 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:14:18.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:14:17.043498+0000 mgr.y (mgr.14144) 244 : cephadm [INF] Saving service iscsi.foo spec with placement count:1 2024-09-17T07:14:18.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:17.967048+0000 mon.a (mon.0) 552 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:18.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:17.971634+0000 mon.a (mon.0) 553 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:18.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:17.972297+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi179.ecjhll", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:14:18.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:17.976410+0000 mon.a (mon.0) 555 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi179.ecjhll", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T07:14:18.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:17.981008+0000 mon.a (mon.0) 556 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:18.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:17.981673+0000 mon.a (mon.0) 557 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:14:18.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:17.043006+0000 mgr.y (mgr.14144) 243 : audit [DBG] from='client.24592 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:14:18.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:14:17.043498+0000 mgr.y (mgr.14144) 244 : cephadm [INF] Saving service iscsi.foo spec with placement count:1 2024-09-17T07:14:18.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:17.967048+0000 mon.a (mon.0) 552 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:18.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:17.971634+0000 mon.a (mon.0) 553 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:18.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:17.972297+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi179.ecjhll", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:14:18.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:17.976410+0000 mon.a (mon.0) 555 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi179.ecjhll", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T07:14:18.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:17.981008+0000 mon.a (mon.0) 556 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:18.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:17.981673+0000 mon.a (mon.0) 557 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:14:18.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:18 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:17.043006+0000 mgr.y (mgr.14144) 243 : audit [DBG] from='client.24592 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:14:18.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:18 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:14:17.043498+0000 mgr.y (mgr.14144) 244 : cephadm [INF] Saving service iscsi.foo spec with placement count:1 2024-09-17T07:14:18.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:18 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:17.967048+0000 mon.a (mon.0) 552 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:18.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:18 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:17.971634+0000 mon.a (mon.0) 553 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:18.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:18 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:17.972297+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi179.ecjhll", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:14:18.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:18 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:17.976410+0000 mon.a (mon.0) 555 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi179.ecjhll", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T07:14:18.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:18 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:17.981008+0000 mon.a (mon.0) 556 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:18.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:18 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:17.981673+0000 mon.a (mon.0) 557 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:14:19.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:17.790320+0000 mgr.y (mgr.14144) 245 : cluster [DBG] pgmap v222: 161 pgs: 161 active+clean; 27 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 430 B/s wr, 171 op/s 2024-09-17T07:14:19.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:14:17.968353+0000 mgr.y (mgr.14144) 246 : cephadm [INF] Saving service rgw.smpl spec with placement count:2 2024-09-17T07:14:19.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:14:17.982317+0000 mgr.y (mgr.14144) 247 : cephadm [INF] Deploying daemon rgw.smpl.smithi179.ecjhll on smithi179 2024-09-17T07:14:19.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:17.790320+0000 mgr.y (mgr.14144) 245 : cluster [DBG] pgmap v222: 161 pgs: 161 active+clean; 27 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 430 B/s wr, 171 op/s 2024-09-17T07:14:19.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:14:17.968353+0000 mgr.y (mgr.14144) 246 : cephadm [INF] Saving service rgw.smpl spec with placement count:2 2024-09-17T07:14:19.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:14:17.982317+0000 mgr.y (mgr.14144) 247 : cephadm [INF] Deploying daemon rgw.smpl.smithi179.ecjhll on smithi179 2024-09-17T07:14:19.826 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:19 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:17.790320+0000 mgr.y (mgr.14144) 245 : cluster [DBG] pgmap v222: 161 pgs: 161 active+clean; 27 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 430 B/s wr, 171 op/s 2024-09-17T07:14:19.826 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:19 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:14:17.968353+0000 mgr.y (mgr.14144) 246 : cephadm [INF] Saving service rgw.smpl spec with placement count:2 2024-09-17T07:14:19.826 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:19 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:14:17.982317+0000 mgr.y (mgr.14144) 247 : cephadm [INF] Deploying daemon rgw.smpl.smithi179.ecjhll on smithi179 2024-09-17T07:14:20.909 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:19.652664+0000 mon.a (mon.0) 558 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2024-09-17T07:14:20.909 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:19.790894+0000 mgr.y (mgr.14144) 248 : cluster [DBG] pgmap v224: 161 pgs: 161 active+clean; 27 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 104 KiB/s rd, 383 B/s wr, 212 op/s 2024-09-17T07:14:20.909 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:20.397401+0000 mon.a (mon.0) 559 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:20.909 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:20.398527+0000 mon.a (mon.0) 560 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi044.vnufiz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:14:20.909 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:20.402016+0000 mon.a (mon.0) 561 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi044.vnufiz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T07:14:20.910 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:20.407320+0000 mon.a (mon.0) 562 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:20.910 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:20.408090+0000 mon.a (mon.0) 563 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:14:20.910 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:14:20.408716+0000 mgr.y (mgr.14144) 249 : cephadm [INF] Deploying daemon rgw.smpl.smithi044.vnufiz on smithi044 2024-09-17T07:14:20.910 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:19.652664+0000 mon.a (mon.0) 558 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2024-09-17T07:14:20.910 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:19.790894+0000 mgr.y (mgr.14144) 248 : cluster [DBG] pgmap v224: 161 pgs: 161 active+clean; 27 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 104 KiB/s rd, 383 B/s wr, 212 op/s 2024-09-17T07:14:20.910 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:20.397401+0000 mon.a (mon.0) 559 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:20.910 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:20.398527+0000 mon.a (mon.0) 560 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi044.vnufiz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:14:20.910 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:20.402016+0000 mon.a (mon.0) 561 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi044.vnufiz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T07:14:20.910 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:20.407320+0000 mon.a (mon.0) 562 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:20.910 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:20.408090+0000 mon.a (mon.0) 563 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:14:20.910 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:14:20.408716+0000 mgr.y (mgr.14144) 249 : cephadm [INF] Deploying daemon rgw.smpl.smithi044.vnufiz on smithi044 2024-09-17T07:14:21.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:19.652664+0000 mon.a (mon.0) 558 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2024-09-17T07:14:21.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:19.790894+0000 mgr.y (mgr.14144) 248 : cluster [DBG] pgmap v224: 161 pgs: 161 active+clean; 27 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 104 KiB/s rd, 383 B/s wr, 212 op/s 2024-09-17T07:14:21.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:20.397401+0000 mon.a (mon.0) 559 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:21.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:20.398527+0000 mon.a (mon.0) 560 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi044.vnufiz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:14:21.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:20.402016+0000 mon.a (mon.0) 561 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi044.vnufiz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T07:14:21.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:20.407320+0000 mon.a (mon.0) 562 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:21.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:20.408090+0000 mon.a (mon.0) 563 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:14:21.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:14:20.408716+0000 mgr.y (mgr.14144) 249 : cephadm [INF] Deploying daemon rgw.smpl.smithi044.vnufiz on smithi044 2024-09-17T07:14:23.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:22 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:21.791551+0000 mgr.y (mgr.14144) 250 : cluster [DBG] pgmap v225: 161 pgs: 161 active+clean; 27 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 270 KiB/s rd, 448 B/s wr, 507 op/s 2024-09-17T07:14:23.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:22 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:22.828326+0000 mon.a (mon.0) 564 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:23.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:22 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:22.829058+0000 mon.a (mon.0) 565 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:14:23.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:22 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:21.791551+0000 mgr.y (mgr.14144) 250 : cluster [DBG] pgmap v225: 161 pgs: 161 active+clean; 27 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 270 KiB/s rd, 448 B/s wr, 507 op/s 2024-09-17T07:14:23.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:22 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:22.828326+0000 mon.a (mon.0) 564 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:23.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:22 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:22.829058+0000 mon.a (mon.0) 565 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:14:23.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:21.791551+0000 mgr.y (mgr.14144) 250 : cluster [DBG] pgmap v225: 161 pgs: 161 active+clean; 27 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 270 KiB/s rd, 448 B/s wr, 507 op/s 2024-09-17T07:14:23.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:22.828326+0000 mon.a (mon.0) 564 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:23.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:22.829058+0000 mon.a (mon.0) 565 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:14:25.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:24 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:23.792387+0000 mgr.y (mgr.14144) 251 : cluster [DBG] pgmap v226: 161 pgs: 161 active+clean; 27 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 314 KiB/s rd, 409 B/s wr, 582 op/s 2024-09-17T07:14:25.136 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:23.792387+0000 mgr.y (mgr.14144) 251 : cluster [DBG] pgmap v226: 161 pgs: 161 active+clean; 27 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 314 KiB/s rd, 409 B/s wr, 582 op/s 2024-09-17T07:14:25.136 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:23.792387+0000 mgr.y (mgr.14144) 251 : cluster [DBG] pgmap v226: 161 pgs: 161 active+clean; 27 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 314 KiB/s rd, 409 B/s wr, 582 op/s 2024-09-17T07:14:27.010 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:25.683351+0000 mon.a (mon.0) 566 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:27.011 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:25.792894+0000 mgr.y (mgr.14144) 252 : cluster [DBG] pgmap v227: 161 pgs: 161 active+clean; 27 KiB data, 78 MiB used, 715 GiB / 715 GiB avail; 363 KiB/s rd, 409 B/s wr, 667 op/s 2024-09-17T07:14:27.011 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:26.444289+0000 mon.a (mon.0) 567 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:27.011 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:26.450938+0000 mon.a (mon.0) 568 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:27.011 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:14:26.453667+0000 mgr.y (mgr.14144) 253 : cephadm [INF] Checking pool "foo" exists for service iscsi.foo 2024-09-17T07:14:27.011 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:26.460164+0000 mon.a (mon.0) 569 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi044.elmtfk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T07:14:27.011 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:26.464526+0000 mon.a (mon.0) 570 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi044.elmtfk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-17T07:14:27.011 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:26.468784+0000 mon.a (mon.0) 571 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:14:27.011 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:14:26.469482+0000 mgr.y (mgr.14144) 254 : cephadm [INF] Deploying daemon iscsi.foo.smithi044.elmtfk on smithi044 2024-09-17T07:14:27.011 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:25.683351+0000 mon.a (mon.0) 566 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:27.012 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:25.792894+0000 mgr.y (mgr.14144) 252 : cluster [DBG] pgmap v227: 161 pgs: 161 active+clean; 27 KiB data, 78 MiB used, 715 GiB / 715 GiB avail; 363 KiB/s rd, 409 B/s wr, 667 op/s 2024-09-17T07:14:27.012 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:26.444289+0000 mon.a (mon.0) 567 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:27.012 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:26.450938+0000 mon.a (mon.0) 568 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:27.012 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:14:26.453667+0000 mgr.y (mgr.14144) 253 : cephadm [INF] Checking pool "foo" exists for service iscsi.foo 2024-09-17T07:14:27.012 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:26.460164+0000 mon.a (mon.0) 569 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi044.elmtfk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T07:14:27.012 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:26.464526+0000 mon.a (mon.0) 570 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi044.elmtfk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-17T07:14:27.012 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:26.468784+0000 mon.a (mon.0) 571 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:14:27.012 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:14:26.469482+0000 mgr.y (mgr.14144) 254 : cephadm [INF] Deploying daemon iscsi.foo.smithi044.elmtfk on smithi044 2024-09-17T07:14:27.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:26 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:25.683351+0000 mon.a (mon.0) 566 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:27.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:26 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:25.792894+0000 mgr.y (mgr.14144) 252 : cluster [DBG] pgmap v227: 161 pgs: 161 active+clean; 27 KiB data, 78 MiB used, 715 GiB / 715 GiB avail; 363 KiB/s rd, 409 B/s wr, 667 op/s 2024-09-17T07:14:27.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:26 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:26.444289+0000 mon.a (mon.0) 567 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:27.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:26 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:26.450938+0000 mon.a (mon.0) 568 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:27.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:26 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:14:26.453667+0000 mgr.y (mgr.14144) 253 : cephadm [INF] Checking pool "foo" exists for service iscsi.foo 2024-09-17T07:14:27.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:26 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:26.460164+0000 mon.a (mon.0) 569 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi044.elmtfk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T07:14:27.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:26 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:26.464526+0000 mon.a (mon.0) 570 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi044.elmtfk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-17T07:14:27.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:26 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:26.468784+0000 mon.a (mon.0) 571 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:14:27.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:26 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:14:26.469482+0000 mgr.y (mgr.14144) 254 : cephadm [INF] Deploying daemon iscsi.foo.smithi044.elmtfk on smithi044 2024-09-17T07:14:29.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:28 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:27.793620+0000 mgr.y (mgr.14144) 255 : cluster [DBG] pgmap v228: 161 pgs: 161 active+clean; 27 KiB data, 87 MiB used, 715 GiB / 715 GiB avail; 377 KiB/s rd, 307 B/s wr, 673 op/s 2024-09-17T07:14:29.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:28 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:27.878979+0000 mon.a (mon.0) 572 : cluster [DBG] mgrmap e15: y(active, since 5m), standbys: x 2024-09-17T07:14:29.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:28 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:27.793620+0000 mgr.y (mgr.14144) 255 : cluster [DBG] pgmap v228: 161 pgs: 161 active+clean; 27 KiB data, 87 MiB used, 715 GiB / 715 GiB avail; 377 KiB/s rd, 307 B/s wr, 673 op/s 2024-09-17T07:14:29.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:28 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:27.878979+0000 mon.a (mon.0) 572 : cluster [DBG] mgrmap e15: y(active, since 5m), standbys: x 2024-09-17T07:14:29.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:28 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:27.793620+0000 mgr.y (mgr.14144) 255 : cluster [DBG] pgmap v228: 161 pgs: 161 active+clean; 27 KiB data, 87 MiB used, 715 GiB / 715 GiB avail; 377 KiB/s rd, 307 B/s wr, 673 op/s 2024-09-17T07:14:29.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:28 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:27.878979+0000 mon.a (mon.0) 572 : cluster [DBG] mgrmap e15: y(active, since 5m), standbys: x 2024-09-17T07:14:30.764 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:29.495877+0000 mon.a (mon.0) 573 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:30.764 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:29.496727+0000 mon.a (mon.0) 574 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:14:30.764 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:29.939164+0000 mon.a (mon.0) 575 : audit [DBG] from='client.? 172.21.15.44:0/3616041418' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:14:30.764 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:30.238884+0000 mon.a (mon.0) 576 : audit [INF] from='client.? 172.21.15.44:0/2702897855' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/367718499"}]: access denied 2024-09-17T07:14:30.764 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:29.495877+0000 mon.a (mon.0) 573 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:30.764 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:29.496727+0000 mon.a (mon.0) 574 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:14:30.764 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:29.939164+0000 mon.a (mon.0) 575 : audit [DBG] from='client.? 172.21.15.44:0/3616041418' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:14:30.765 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:30.238884+0000 mon.a (mon.0) 576 : audit [INF] from='client.? 172.21.15.44:0/2702897855' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/367718499"}]: access denied 2024-09-17T07:14:30.819 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:30 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:29.495877+0000 mon.a (mon.0) 573 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:30.820 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:30 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:29.496727+0000 mon.a (mon.0) 574 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:14:30.820 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:30 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:29.939164+0000 mon.a (mon.0) 575 : audit [DBG] from='client.? 172.21.15.44:0/3616041418' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:14:30.820 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:30 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:30.238884+0000 mon.a (mon.0) 576 : audit [INF] from='client.? 172.21.15.44:0/2702897855' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/367718499"}]: access denied 2024-09-17T07:14:31.770 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:29.794188+0000 mgr.y (mgr.14144) 256 : cluster [DBG] pgmap v229: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 358 KiB/s rd, 302 B/s wr, 631 op/s 2024-09-17T07:14:31.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:29.794188+0000 mgr.y (mgr.14144) 256 : cluster [DBG] pgmap v229: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 358 KiB/s rd, 302 B/s wr, 631 op/s 2024-09-17T07:14:31.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:29.794188+0000 mgr.y (mgr.14144) 256 : cluster [DBG] pgmap v229: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 358 KiB/s rd, 302 B/s wr, 631 op/s 2024-09-17T07:14:33.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:31.794927+0000 mgr.y (mgr.14144) 257 : cluster [DBG] pgmap v230: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 303 KiB/s rd, 341 B/s wr, 534 op/s 2024-09-17T07:14:33.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:32.857217+0000 mon.a (mon.0) 577 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:33.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:31.794927+0000 mgr.y (mgr.14144) 257 : cluster [DBG] pgmap v230: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 303 KiB/s rd, 341 B/s wr, 534 op/s 2024-09-17T07:14:33.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:32.857217+0000 mon.a (mon.0) 577 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:33.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:31.794927+0000 mgr.y (mgr.14144) 257 : cluster [DBG] pgmap v230: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 303 KiB/s rd, 341 B/s wr, 534 op/s 2024-09-17T07:14:33.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:32.857217+0000 mon.a (mon.0) 577 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:35.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:33.795369+0000 mgr.y (mgr.14144) 258 : cluster [DBG] pgmap v231: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 166 KiB/s rd, 170 B/s wr, 289 op/s 2024-09-17T07:14:35.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:33.937982+0000 mon.a (mon.0) 578 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:35.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:33.938778+0000 mgr.y (mgr.14144) 259 : cluster [DBG] pgmap v232: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 131 KiB/s rd, 201 B/s wr, 225 op/s 2024-09-17T07:14:35.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:33.942029+0000 mon.a (mon.0) 579 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:35.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:33.946646+0000 mon.a (mon.0) 580 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T07:14:35.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:33.947095+0000 mgr.y (mgr.14144) 260 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T07:14:35.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:14:33.948875+0000 mgr.y (mgr.14144) 261 : cephadm [INF] Adding iSCSI gateway http://:@172.21.15.44:5000 to Dashboard 2024-09-17T07:14:35.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:33.949055+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T07:14:35.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:33.949317+0000 mgr.y (mgr.14144) 262 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T07:14:35.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:33.955327+0000 mon.a (mon.0) 582 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:35.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:33.957451+0000 mon.a (mon.0) 583 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi044"}]: dispatch 2024-09-17T07:14:35.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:33.957884+0000 mgr.y (mgr.14144) 263 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi044"}]: dispatch 2024-09-17T07:14:35.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:33.966440+0000 mon.a (mon.0) 584 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:35.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:33.967309+0000 mon.a (mon.0) 585 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:14:35.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:34.503814+0000 mon.a (mon.0) 586 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T07:14:35.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:33.795369+0000 mgr.y (mgr.14144) 258 : cluster [DBG] pgmap v231: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 166 KiB/s rd, 170 B/s wr, 289 op/s 2024-09-17T07:14:35.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:33.937982+0000 mon.a (mon.0) 578 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:35.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:33.938778+0000 mgr.y (mgr.14144) 259 : cluster [DBG] pgmap v232: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 131 KiB/s rd, 201 B/s wr, 225 op/s 2024-09-17T07:14:35.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:33.942029+0000 mon.a (mon.0) 579 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:35.327 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:33.946646+0000 mon.a (mon.0) 580 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T07:14:35.327 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:33.947095+0000 mgr.y (mgr.14144) 260 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T07:14:35.327 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:14:33.948875+0000 mgr.y (mgr.14144) 261 : cephadm [INF] Adding iSCSI gateway http://:@172.21.15.44:5000 to Dashboard 2024-09-17T07:14:35.327 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:33.949055+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T07:14:35.327 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:33.949317+0000 mgr.y (mgr.14144) 262 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T07:14:35.327 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:33.955327+0000 mon.a (mon.0) 582 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:35.327 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:33.957451+0000 mon.a (mon.0) 583 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi044"}]: dispatch 2024-09-17T07:14:35.327 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:33.957884+0000 mgr.y (mgr.14144) 263 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi044"}]: dispatch 2024-09-17T07:14:35.327 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:33.966440+0000 mon.a (mon.0) 584 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:35.327 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:33.967309+0000 mon.a (mon.0) 585 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:14:35.327 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:34.503814+0000 mon.a (mon.0) 586 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T07:14:35.362 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:33.795369+0000 mgr.y (mgr.14144) 258 : cluster [DBG] pgmap v231: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 166 KiB/s rd, 170 B/s wr, 289 op/s 2024-09-17T07:14:35.362 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:33.937982+0000 mon.a (mon.0) 578 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:35.362 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:33.938778+0000 mgr.y (mgr.14144) 259 : cluster [DBG] pgmap v232: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 131 KiB/s rd, 201 B/s wr, 225 op/s 2024-09-17T07:14:35.362 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:33.942029+0000 mon.a (mon.0) 579 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:35.362 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:33.946646+0000 mon.a (mon.0) 580 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T07:14:35.362 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:33.947095+0000 mgr.y (mgr.14144) 260 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T07:14:35.362 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:14:33.948875+0000 mgr.y (mgr.14144) 261 : cephadm [INF] Adding iSCSI gateway http://:@172.21.15.44:5000 to Dashboard 2024-09-17T07:14:35.362 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:33.949055+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T07:14:35.363 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:33.949317+0000 mgr.y (mgr.14144) 262 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T07:14:35.363 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:33.955327+0000 mon.a (mon.0) 582 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:35.363 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:33.957451+0000 mon.a (mon.0) 583 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi044"}]: dispatch 2024-09-17T07:14:35.363 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:33.957884+0000 mgr.y (mgr.14144) 263 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi044"}]: dispatch 2024-09-17T07:14:35.363 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:33.966440+0000 mon.a (mon.0) 584 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:35.363 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:33.967309+0000 mon.a (mon.0) 585 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:14:35.363 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:34.503814+0000 mon.a (mon.0) 586 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T07:14:37.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:35.939328+0000 mgr.y (mgr.14144) 264 : cluster [DBG] pgmap v233: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 201 B/s wr, 133 op/s 2024-09-17T07:14:37.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:36.775325+0000 mon.a (mon.0) 587 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:37.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:35.939328+0000 mgr.y (mgr.14144) 264 : cluster [DBG] pgmap v233: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 201 B/s wr, 133 op/s 2024-09-17T07:14:37.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:36.775325+0000 mon.a (mon.0) 587 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:37.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:35.939328+0000 mgr.y (mgr.14144) 264 : cluster [DBG] pgmap v233: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 201 B/s wr, 133 op/s 2024-09-17T07:14:37.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:36.775325+0000 mon.a (mon.0) 587 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:38.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:37.296775+0000 mon.a (mon.0) 588 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:38.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:37.302833+0000 mon.a (mon.0) 589 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:38.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:37.296775+0000 mon.a (mon.0) 588 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:38.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:37.302833+0000 mon.a (mon.0) 589 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:38.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:38 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:37.296775+0000 mon.a (mon.0) 588 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:38.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:38 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:37.302833+0000 mon.a (mon.0) 589 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:14:39.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:37.940109+0000 mgr.y (mgr.14144) 265 : cluster [DBG] pgmap v234: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 9.0 KiB/s rd, 100 B/s wr, 15 op/s 2024-09-17T07:14:39.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:37.940109+0000 mgr.y (mgr.14144) 265 : cluster [DBG] pgmap v234: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 9.0 KiB/s rd, 100 B/s wr, 15 op/s 2024-09-17T07:14:39.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:37.940109+0000 mgr.y (mgr.14144) 265 : cluster [DBG] pgmap v234: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 9.0 KiB/s rd, 100 B/s wr, 15 op/s 2024-09-17T07:14:41.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:39.940654+0000 mgr.y (mgr.14144) 266 : cluster [DBG] pgmap v235: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 100 B/s rd, 100 B/s wr, 0 op/s 2024-09-17T07:14:41.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:39.940654+0000 mgr.y (mgr.14144) 266 : cluster [DBG] pgmap v235: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 100 B/s rd, 100 B/s wr, 0 op/s 2024-09-17T07:14:41.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:40 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:39.940654+0000 mgr.y (mgr.14144) 266 : cluster [DBG] pgmap v235: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 100 B/s rd, 100 B/s wr, 0 op/s 2024-09-17T07:14:43.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:41.941438+0000 mgr.y (mgr.14144) 267 : cluster [DBG] pgmap v236: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:14:43.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:41.941438+0000 mgr.y (mgr.14144) 267 : cluster [DBG] pgmap v236: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:14:43.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:41.941438+0000 mgr.y (mgr.14144) 267 : cluster [DBG] pgmap v236: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:14:44.310 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:43.179765+0000 mon.a (mon.0) 590 : audit [DBG] from='client.? 172.21.15.44:0/702174376' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:14:44.310 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:43.469788+0000 mon.c (mon.1) 23 : audit [INF] from='client.? 172.21.15.44:0/1802581942' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/367718499"}]: access denied 2024-09-17T07:14:44.310 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:43.179765+0000 mon.a (mon.0) 590 : audit [DBG] from='client.? 172.21.15.44:0/702174376' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:14:44.310 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:43.469788+0000 mon.c (mon.1) 23 : audit [INF] from='client.? 172.21.15.44:0/1802581942' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/367718499"}]: access denied 2024-09-17T07:14:44.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:43.179765+0000 mon.a (mon.0) 590 : audit [DBG] from='client.? 172.21.15.44:0/702174376' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:14:44.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:43.469788+0000 mon.c (mon.1) 23 : audit [INF] from='client.? 172.21.15.44:0/1802581942' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/367718499"}]: access denied 2024-09-17T07:14:45.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:43.942267+0000 mgr.y (mgr.14144) 268 : cluster [DBG] pgmap v237: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:14:45.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:43.942267+0000 mgr.y (mgr.14144) 268 : cluster [DBG] pgmap v237: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:14:45.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:43.942267+0000 mgr.y (mgr.14144) 268 : cluster [DBG] pgmap v237: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:14:47.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:45.943067+0000 mgr.y (mgr.14144) 269 : cluster [DBG] pgmap v238: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:14:47.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:45.943067+0000 mgr.y (mgr.14144) 269 : cluster [DBG] pgmap v238: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:14:47.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:45.943067+0000 mgr.y (mgr.14144) 269 : cluster [DBG] pgmap v238: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:14:48.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:47.852520+0000 mon.a (mon.0) 591 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-09-17T07:14:48.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:47.852796+0000 mon.a (mon.0) 592 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-09-17T07:14:48.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:47.853013+0000 mon.a (mon.0) 593 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-09-17T07:14:48.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:47.870236+0000 mon.a (mon.0) 594 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:14:48.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:47.871395+0000 mon.a (mon.0) 595 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:14:48.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:47.852520+0000 mon.a (mon.0) 591 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-09-17T07:14:48.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:47.852796+0000 mon.a (mon.0) 592 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-09-17T07:14:48.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:47.853013+0000 mon.a (mon.0) 593 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-09-17T07:14:48.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:47.870236+0000 mon.a (mon.0) 594 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:14:48.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:47.871395+0000 mon.a (mon.0) 595 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:14:48.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:47.852520+0000 mon.a (mon.0) 591 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-09-17T07:14:48.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:47.852796+0000 mon.a (mon.0) 592 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-09-17T07:14:48.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:47.853013+0000 mon.a (mon.0) 593 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-09-17T07:14:48.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:47.870236+0000 mon.a (mon.0) 594 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:14:48.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:47.871395+0000 mon.a (mon.0) 595 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:14:49.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:47.943665+0000 mgr.y (mgr.14144) 270 : cluster [DBG] pgmap v239: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:14:49.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:48.020484+0000 mon.a (mon.0) 596 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-09-17T07:14:49.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:48.020599+0000 mon.a (mon.0) 597 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-17T07:14:49.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:48.020685+0000 mon.a (mon.0) 598 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-09-17T07:14:49.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:48.020770+0000 mon.a (mon.0) 599 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-09-17T07:14:49.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:47.943665+0000 mgr.y (mgr.14144) 270 : cluster [DBG] pgmap v239: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:14:49.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:48.020484+0000 mon.a (mon.0) 596 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-09-17T07:14:49.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:48.020599+0000 mon.a (mon.0) 597 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-17T07:14:49.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:48.020685+0000 mon.a (mon.0) 598 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-09-17T07:14:49.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:48.020770+0000 mon.a (mon.0) 599 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-09-17T07:14:49.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:47.943665+0000 mgr.y (mgr.14144) 270 : cluster [DBG] pgmap v239: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:14:49.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:48.020484+0000 mon.a (mon.0) 596 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-09-17T07:14:49.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:48.020599+0000 mon.a (mon.0) 597 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-17T07:14:49.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:48.020685+0000 mon.a (mon.0) 598 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-09-17T07:14:49.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:48.020770+0000 mon.a (mon.0) 599 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-09-17T07:14:50.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:49.035593+0000 mon.a (mon.0) 600 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-09-17T07:14:50.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:49.035593+0000 mon.a (mon.0) 600 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-09-17T07:14:50.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:49.035593+0000 mon.a (mon.0) 600 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-09-17T07:14:51.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:49.944168+0000 mgr.y (mgr.14144) 271 : cluster [DBG] pgmap v242: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-09-17T07:14:51.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:49.944168+0000 mgr.y (mgr.14144) 271 : cluster [DBG] pgmap v242: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-09-17T07:14:51.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:49.944168+0000 mgr.y (mgr.14144) 271 : cluster [DBG] pgmap v242: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-09-17T07:14:53.324 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:53 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:51.944973+0000 mgr.y (mgr.14144) 272 : cluster [DBG] pgmap v243: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T07:14:53.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:53 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:51.944973+0000 mgr.y (mgr.14144) 272 : cluster [DBG] pgmap v243: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T07:14:53.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:51.944973+0000 mgr.y (mgr.14144) 272 : cluster [DBG] pgmap v243: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T07:14:55.299 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:53.945530+0000 mgr.y (mgr.14144) 273 : cluster [DBG] pgmap v244: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T07:14:55.300 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:53.945530+0000 mgr.y (mgr.14144) 273 : cluster [DBG] pgmap v244: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T07:14:55.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:53.945530+0000 mgr.y (mgr.14144) 273 : cluster [DBG] pgmap v244: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T07:14:57.210 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:55.946109+0000 mgr.y (mgr.14144) 274 : cluster [DBG] pgmap v245: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-17T07:14:57.210 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:56.582977+0000 mon.a (mon.0) 601 : audit [DBG] from='client.? 172.21.15.44:0/1370859823' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:14:57.210 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:14:56.874895+0000 mon.c (mon.1) 24 : audit [INF] from='client.? 172.21.15.44:0/1598183955' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/367718499"}]: access denied 2024-09-17T07:14:57.210 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:55.946109+0000 mgr.y (mgr.14144) 274 : cluster [DBG] pgmap v245: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-17T07:14:57.210 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:56.582977+0000 mon.a (mon.0) 601 : audit [DBG] from='client.? 172.21.15.44:0/1370859823' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:14:57.210 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:14:56.874895+0000 mon.c (mon.1) 24 : audit [INF] from='client.? 172.21.15.44:0/1598183955' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/367718499"}]: access denied 2024-09-17T07:14:57.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:55.946109+0000 mgr.y (mgr.14144) 274 : cluster [DBG] pgmap v245: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-17T07:14:57.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:56.582977+0000 mon.a (mon.0) 601 : audit [DBG] from='client.? 172.21.15.44:0/1370859823' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:14:57.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:14:56.874895+0000 mon.c (mon.1) 24 : audit [INF] from='client.? 172.21.15.44:0/1598183955' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/367718499"}]: access denied 2024-09-17T07:14:59.324 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:14:59 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:57.946979+0000 mgr.y (mgr.14144) 275 : cluster [DBG] pgmap v246: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 103 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T07:14:59.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:14:59 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:57.946979+0000 mgr.y (mgr.14144) 275 : cluster [DBG] pgmap v246: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 103 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T07:14:59.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:14:59 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:57.946979+0000 mgr.y (mgr.14144) 275 : cluster [DBG] pgmap v246: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 103 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T07:15:01.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:14:59.947625+0000 mgr.y (mgr.14144) 276 : cluster [DBG] pgmap v247: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 93 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T07:15:01.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:14:59.947625+0000 mgr.y (mgr.14144) 276 : cluster [DBG] pgmap v247: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 93 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T07:15:01.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:00 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:14:59.947625+0000 mgr.y (mgr.14144) 276 : cluster [DBG] pgmap v247: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 93 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T07:15:03.280 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:03 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:15:01.948468+0000 mgr.y (mgr.14144) 277 : cluster [DBG] pgmap v248: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T07:15:03.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:03 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:15:01.948468+0000 mgr.y (mgr.14144) 277 : cluster [DBG] pgmap v248: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T07:15:03.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:03 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:15:01.948468+0000 mgr.y (mgr.14144) 277 : cluster [DBG] pgmap v248: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T07:15:05.324 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:15:03.949157+0000 mgr.y (mgr.14144) 278 : cluster [DBG] pgmap v249: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:05.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:15:03.949157+0000 mgr.y (mgr.14144) 278 : cluster [DBG] pgmap v249: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:05.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:05 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:15:03.949157+0000 mgr.y (mgr.14144) 278 : cluster [DBG] pgmap v249: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:07.324 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:15:05.949737+0000 mgr.y (mgr.14144) 279 : cluster [DBG] pgmap v250: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:07.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:15:05.949737+0000 mgr.y (mgr.14144) 279 : cluster [DBG] pgmap v250: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:07.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:15:05.949737+0000 mgr.y (mgr.14144) 279 : cluster [DBG] pgmap v250: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:09.150 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:15:07.950369+0000 mgr.y (mgr.14144) 280 : cluster [DBG] pgmap v251: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:09.151 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:15:07.950369+0000 mgr.y (mgr.14144) 280 : cluster [DBG] pgmap v251: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:09.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:15:07.950369+0000 mgr.y (mgr.14144) 280 : cluster [DBG] pgmap v251: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:10.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:15:09.901304+0000 mon.c (mon.1) 25 : audit [DBG] from='client.? 172.21.15.44:0/1527714744' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:15:10.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:15:09.950814+0000 mgr.y (mgr.14144) 281 : cluster [DBG] pgmap v252: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:10.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:15:10.193451+0000 mon.c (mon.1) 26 : audit [INF] from='client.? 172.21.15.44:0/1733081432' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/367718499"}]: access denied 2024-09-17T07:15:10.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:15:09.901304+0000 mon.c (mon.1) 25 : audit [DBG] from='client.? 172.21.15.44:0/1527714744' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:15:10.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:15:09.950814+0000 mgr.y (mgr.14144) 281 : cluster [DBG] pgmap v252: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:10.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:15:10.193451+0000 mon.c (mon.1) 26 : audit [INF] from='client.? 172.21.15.44:0/1733081432' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/367718499"}]: access denied 2024-09-17T07:15:11.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:15:09.901304+0000 mon.c (mon.1) 25 : audit [DBG] from='client.? 172.21.15.44:0/1527714744' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:15:11.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:15:09.950814+0000 mgr.y (mgr.14144) 281 : cluster [DBG] pgmap v252: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:11.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:15:10.193451+0000 mon.c (mon.1) 26 : audit [INF] from='client.? 172.21.15.44:0/1733081432' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/367718499"}]: access denied 2024-09-17T07:15:13.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:13 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:15:11.951626+0000 mgr.y (mgr.14144) 282 : cluster [DBG] pgmap v253: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:13.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:15:11.951626+0000 mgr.y (mgr.14144) 282 : cluster [DBG] pgmap v253: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:13.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:15:11.951626+0000 mgr.y (mgr.14144) 282 : cluster [DBG] pgmap v253: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:15.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:15:13.952257+0000 mgr.y (mgr.14144) 283 : cluster [DBG] pgmap v254: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:15.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:15:13.952257+0000 mgr.y (mgr.14144) 283 : cluster [DBG] pgmap v254: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:15.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:15:13.952257+0000 mgr.y (mgr.14144) 283 : cluster [DBG] pgmap v254: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:17.324 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:17 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:15:15.952720+0000 mgr.y (mgr.14144) 284 : cluster [DBG] pgmap v255: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:17.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:17 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:15:15.952720+0000 mgr.y (mgr.14144) 284 : cluster [DBG] pgmap v255: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:17.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:15:15.952720+0000 mgr.y (mgr.14144) 284 : cluster [DBG] pgmap v255: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:19.324 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:15:17.953422+0000 mgr.y (mgr.14144) 285 : cluster [DBG] pgmap v256: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:19.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:15:17.953422+0000 mgr.y (mgr.14144) 285 : cluster [DBG] pgmap v256: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:19.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:19 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:15:17.953422+0000 mgr.y (mgr.14144) 285 : cluster [DBG] pgmap v256: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:21.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:15:19.954033+0000 mgr.y (mgr.14144) 286 : cluster [DBG] pgmap v257: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:21.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:15:19.954033+0000 mgr.y (mgr.14144) 286 : cluster [DBG] pgmap v257: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:21.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:15:19.954033+0000 mgr.y (mgr.14144) 286 : cluster [DBG] pgmap v257: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:23.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:23 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:15:21.954771+0000 mgr.y (mgr.14144) 287 : cluster [DBG] pgmap v258: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:23.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:23 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:15:21.954771+0000 mgr.y (mgr.14144) 287 : cluster [DBG] pgmap v258: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:23.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:15:21.954771+0000 mgr.y (mgr.14144) 287 : cluster [DBG] pgmap v258: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:24.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:15:23.384002+0000 mon.c (mon.1) 27 : audit [DBG] from='client.? 172.21.15.44:0/1165724918' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:15:24.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:15:23.678888+0000 mon.a (mon.0) 602 : audit [INF] from='client.? 172.21.15.44:0/476805482' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/367718499"}]: access denied 2024-09-17T07:15:24.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:15:23.384002+0000 mon.c (mon.1) 27 : audit [DBG] from='client.? 172.21.15.44:0/1165724918' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:15:24.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:15:23.678888+0000 mon.a (mon.0) 602 : audit [INF] from='client.? 172.21.15.44:0/476805482' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/367718499"}]: access denied 2024-09-17T07:15:24.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:24 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:15:23.384002+0000 mon.c (mon.1) 27 : audit [DBG] from='client.? 172.21.15.44:0/1165724918' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:15:24.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:24 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:15:23.678888+0000 mon.a (mon.0) 602 : audit [INF] from='client.? 172.21.15.44:0/476805482' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/367718499"}]: access denied 2024-09-17T07:15:25.276 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:15:23.955297+0000 mgr.y (mgr.14144) 288 : cluster [DBG] pgmap v259: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:25.277 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:15:23.955297+0000 mgr.y (mgr.14144) 288 : cluster [DBG] pgmap v259: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:25.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:25 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:15:23.955297+0000 mgr.y (mgr.14144) 288 : cluster [DBG] pgmap v259: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:27.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:15:25.956069+0000 mgr.y (mgr.14144) 289 : cluster [DBG] pgmap v260: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:27.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:15:25.956069+0000 mgr.y (mgr.14144) 289 : cluster [DBG] pgmap v260: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:27.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:27 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:15:25.956069+0000 mgr.y (mgr.14144) 289 : cluster [DBG] pgmap v260: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:29.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:15:27.956882+0000 mgr.y (mgr.14144) 290 : cluster [DBG] pgmap v261: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:29.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:15:27.956882+0000 mgr.y (mgr.14144) 290 : cluster [DBG] pgmap v261: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:29.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:15:27.956882+0000 mgr.y (mgr.14144) 290 : cluster [DBG] pgmap v261: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:31.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:15:29.957607+0000 mgr.y (mgr.14144) 291 : cluster [DBG] pgmap v262: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:31.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:15:29.957607+0000 mgr.y (mgr.14144) 291 : cluster [DBG] pgmap v262: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:31.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:30 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:15:29.957607+0000 mgr.y (mgr.14144) 291 : cluster [DBG] pgmap v262: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:33.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:15:31.958417+0000 mgr.y (mgr.14144) 292 : cluster [DBG] pgmap v263: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:33.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:15:31.958417+0000 mgr.y (mgr.14144) 292 : cluster [DBG] pgmap v263: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:33.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:15:31.958417+0000 mgr.y (mgr.14144) 292 : cluster [DBG] pgmap v263: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:35.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:15:33.959142+0000 mgr.y (mgr.14144) 293 : cluster [DBG] pgmap v264: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:35.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:15:33.959142+0000 mgr.y (mgr.14144) 293 : cluster [DBG] pgmap v264: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:35.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:35 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:15:33.959142+0000 mgr.y (mgr.14144) 293 : cluster [DBG] pgmap v264: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:37.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:15:35.959870+0000 mgr.y (mgr.14144) 294 : cluster [DBG] pgmap v265: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:37.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:15:35.959870+0000 mgr.y (mgr.14144) 294 : cluster [DBG] pgmap v265: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:37.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:37 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:15:35.959870+0000 mgr.y (mgr.14144) 294 : cluster [DBG] pgmap v265: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:38.288 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:15:37.306566+0000 mon.a (mon.0) 603 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:15:38.289 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:15:37.306566+0000 mon.a (mon.0) 603 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:15:38.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:38 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:15:37.306566+0000 mon.a (mon.0) 603 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:15:39.064 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:15:37.960406+0000 mgr.y (mgr.14144) 295 : cluster [DBG] pgmap v266: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:39.064 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:15:37.960406+0000 mgr.y (mgr.14144) 295 : cluster [DBG] pgmap v266: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:39.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:15:37.960406+0000 mgr.y (mgr.14144) 295 : cluster [DBG] pgmap v266: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:15:41.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:41 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:15:39.960943+0000 mgr.y (mgr.14144) 296 : cluster [DBG] pgmap v267: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:41.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:41 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:15:40.219651+0000 mon.a (mon.0) 604 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:15:41.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:41 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:15:40.439000+0000 mgr.y (mgr.14144) 297 : cluster [DBG] pgmap v268: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:41.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:41 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:15:40.442608+0000 mon.a (mon.0) 605 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:15:41.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:41 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:15:39.960943+0000 mgr.y (mgr.14144) 296 : cluster [DBG] pgmap v267: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:41.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:41 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:15:40.219651+0000 mon.a (mon.0) 604 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:15:41.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:41 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:15:40.439000+0000 mgr.y (mgr.14144) 297 : cluster [DBG] pgmap v268: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:41.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:41 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:15:40.442608+0000 mon.a (mon.0) 605 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:15:41.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:41 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:15:39.960943+0000 mgr.y (mgr.14144) 296 : cluster [DBG] pgmap v267: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:41.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:41 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:15:40.219651+0000 mon.a (mon.0) 604 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:15:41.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:41 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:15:40.439000+0000 mgr.y (mgr.14144) 297 : cluster [DBG] pgmap v268: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:41.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:41 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:15:40.442608+0000 mon.a (mon.0) 605 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:15:43.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:43 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:15:42.439683+0000 mgr.y (mgr.14144) 298 : cluster [DBG] pgmap v269: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:43.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:43 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:15:42.439683+0000 mgr.y (mgr.14144) 298 : cluster [DBG] pgmap v269: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:43.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:43 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:15:42.439683+0000 mgr.y (mgr.14144) 298 : cluster [DBG] pgmap v269: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:45.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:15:44.440403+0000 mgr.y (mgr.14144) 299 : cluster [DBG] pgmap v270: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:45.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:15:44.440403+0000 mgr.y (mgr.14144) 299 : cluster [DBG] pgmap v270: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:45.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:15:44.440403+0000 mgr.y (mgr.14144) 299 : cluster [DBG] pgmap v270: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:47.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:15:46.441316+0000 mgr.y (mgr.14144) 300 : cluster [DBG] pgmap v271: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:47.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:15:46.441316+0000 mgr.y (mgr.14144) 300 : cluster [DBG] pgmap v271: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:47.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:15:46.441316+0000 mgr.y (mgr.14144) 300 : cluster [DBG] pgmap v271: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:48.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:15:47.875145+0000 mon.a (mon.0) 606 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:15:48.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:15:47.875345+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:15:48.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:15:47.875145+0000 mon.a (mon.0) 606 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:15:48.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:15:47.875345+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:15:48.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:15:47.875145+0000 mon.a (mon.0) 606 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:15:48.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:15:47.875345+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:15:49.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:15:48.441899+0000 mgr.y (mgr.14144) 301 : cluster [DBG] pgmap v272: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:49.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:15:48.441899+0000 mgr.y (mgr.14144) 301 : cluster [DBG] pgmap v272: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:49.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:15:48.441899+0000 mgr.y (mgr.14144) 301 : cluster [DBG] pgmap v272: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:51.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:15:50.442435+0000 mgr.y (mgr.14144) 302 : cluster [DBG] pgmap v273: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:51.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:15:50.442435+0000 mgr.y (mgr.14144) 302 : cluster [DBG] pgmap v273: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:51.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:15:50.442435+0000 mgr.y (mgr.14144) 302 : cluster [DBG] pgmap v273: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:53.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:53 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:15:52.443180+0000 mgr.y (mgr.14144) 303 : cluster [DBG] pgmap v274: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:53.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:53 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:15:52.443180+0000 mgr.y (mgr.14144) 303 : cluster [DBG] pgmap v274: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:53.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:15:52.443180+0000 mgr.y (mgr.14144) 303 : cluster [DBG] pgmap v274: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:55.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:15:54.443692+0000 mgr.y (mgr.14144) 304 : cluster [DBG] pgmap v275: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:55.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:15:54.443692+0000 mgr.y (mgr.14144) 304 : cluster [DBG] pgmap v275: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:55.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:15:54.443692+0000 mgr.y (mgr.14144) 304 : cluster [DBG] pgmap v275: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:57.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:15:56.444428+0000 mgr.y (mgr.14144) 305 : cluster [DBG] pgmap v276: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:57.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:15:56.444428+0000 mgr.y (mgr.14144) 305 : cluster [DBG] pgmap v276: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:57.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:15:56.444428+0000 mgr.y (mgr.14144) 305 : cluster [DBG] pgmap v276: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:59.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:15:59 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:15:58.445039+0000 mgr.y (mgr.14144) 306 : cluster [DBG] pgmap v277: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:59.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:15:59 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:15:58.445039+0000 mgr.y (mgr.14144) 306 : cluster [DBG] pgmap v277: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:15:59.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:15:59 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:15:58.445039+0000 mgr.y (mgr.14144) 306 : cluster [DBG] pgmap v277: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:01.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:16:00.445546+0000 mgr.y (mgr.14144) 307 : cluster [DBG] pgmap v278: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:01.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:16:00.445546+0000 mgr.y (mgr.14144) 307 : cluster [DBG] pgmap v278: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:01.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:00 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:16:00.445546+0000 mgr.y (mgr.14144) 307 : cluster [DBG] pgmap v278: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:03.817 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:03 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:16:02.446286+0000 mgr.y (mgr.14144) 308 : cluster [DBG] pgmap v279: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:03.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:03 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:16:02.446286+0000 mgr.y (mgr.14144) 308 : cluster [DBG] pgmap v279: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:03.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:03 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:16:02.446286+0000 mgr.y (mgr.14144) 308 : cluster [DBG] pgmap v279: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:05.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:16:04.446839+0000 mgr.y (mgr.14144) 309 : cluster [DBG] pgmap v280: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:05.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:16:04.446839+0000 mgr.y (mgr.14144) 309 : cluster [DBG] pgmap v280: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:05.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:05 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:16:04.446839+0000 mgr.y (mgr.14144) 309 : cluster [DBG] pgmap v280: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:07.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:16:06.447549+0000 mgr.y (mgr.14144) 310 : cluster [DBG] pgmap v281: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:07.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:16:06.447549+0000 mgr.y (mgr.14144) 310 : cluster [DBG] pgmap v281: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:07.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:16:06.447549+0000 mgr.y (mgr.14144) 310 : cluster [DBG] pgmap v281: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:09.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:16:08.448141+0000 mgr.y (mgr.14144) 311 : cluster [DBG] pgmap v282: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:09.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:16:08.448141+0000 mgr.y (mgr.14144) 311 : cluster [DBG] pgmap v282: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:09.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:16:08.448141+0000 mgr.y (mgr.14144) 311 : cluster [DBG] pgmap v282: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:11.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:16:10.448773+0000 mgr.y (mgr.14144) 312 : cluster [DBG] pgmap v283: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:11.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:16:10.448773+0000 mgr.y (mgr.14144) 312 : cluster [DBG] pgmap v283: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:11.113 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:16:10.448773+0000 mgr.y (mgr.14144) 312 : cluster [DBG] pgmap v283: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:13.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:16:12.449689+0000 mgr.y (mgr.14144) 313 : cluster [DBG] pgmap v284: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:13.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:16:12.449689+0000 mgr.y (mgr.14144) 313 : cluster [DBG] pgmap v284: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:13.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:13 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:16:12.449689+0000 mgr.y (mgr.14144) 313 : cluster [DBG] pgmap v284: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:15.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:16:14.450274+0000 mgr.y (mgr.14144) 314 : cluster [DBG] pgmap v285: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:15.874 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:16:14.450274+0000 mgr.y (mgr.14144) 314 : cluster [DBG] pgmap v285: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:15.875 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:16:14.450274+0000 mgr.y (mgr.14144) 314 : cluster [DBG] pgmap v285: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:17.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:17 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:16:16.451093+0000 mgr.y (mgr.14144) 315 : cluster [DBG] pgmap v286: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:17.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:17 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:16:16.451093+0000 mgr.y (mgr.14144) 315 : cluster [DBG] pgmap v286: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:17.863 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:16:16.451093+0000 mgr.y (mgr.14144) 315 : cluster [DBG] pgmap v286: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:19.786 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:16:18.451809+0000 mgr.y (mgr.14144) 316 : cluster [DBG] pgmap v287: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:19.786 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:16:18.451809+0000 mgr.y (mgr.14144) 316 : cluster [DBG] pgmap v287: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:19.863 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:19 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:16:18.451809+0000 mgr.y (mgr.14144) 316 : cluster [DBG] pgmap v287: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:20.155 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-09-17T07:16:20.789 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:16:20.452256+0000 mgr.y (mgr.14144) 317 : cluster [DBG] pgmap v288: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:20.789 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:16:20.452256+0000 mgr.y (mgr.14144) 317 : cluster [DBG] pgmap v288: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:21.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:16:20.452256+0000 mgr.y (mgr.14144) 317 : cluster [DBG] pgmap v288: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:22.934 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-09-17T07:16:23.236 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:23 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:16:22.089445+0000 mon.a (mon.0) 608 : audit [INF] from='client.? 172.21.15.44:0/3326656807' entity='client.admin' 2024-09-17T07:16:23.236 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:23 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:16:22.452886+0000 mgr.y (mgr.14144) 318 : cluster [DBG] pgmap v289: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:23.236 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:23 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:16:22.089445+0000 mon.a (mon.0) 608 : audit [INF] from='client.? 172.21.15.44:0/3326656807' entity='client.admin' 2024-09-17T07:16:23.236 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:23 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:16:22.452886+0000 mgr.y (mgr.14144) 318 : cluster [DBG] pgmap v289: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:23.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:16:22.089445+0000 mon.a (mon.0) 608 : audit [INF] from='client.? 172.21.15.44:0/3326656807' entity='client.admin' 2024-09-17T07:16:23.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:16:22.452886+0000 mgr.y (mgr.14144) 318 : cluster [DBG] pgmap v289: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:25.534 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'ceph config set global log_to_journald false --force' 2024-09-17T07:16:25.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:16:24.453333+0000 mgr.y (mgr.14144) 319 : cluster [DBG] pgmap v290: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:25.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:16:24.877182+0000 mon.a (mon.0) 609 : audit [INF] from='client.? 172.21.15.44:0/2821758756' entity='client.admin' 2024-09-17T07:16:25.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:16:24.453333+0000 mgr.y (mgr.14144) 319 : cluster [DBG] pgmap v290: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:25.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:16:24.877182+0000 mon.a (mon.0) 609 : audit [INF] from='client.? 172.21.15.44:0/2821758756' entity='client.admin' 2024-09-17T07:16:25.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:25 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:16:24.453333+0000 mgr.y (mgr.14144) 319 : cluster [DBG] pgmap v290: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:25.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:25 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:16:24.877182+0000 mon.a (mon.0) 609 : audit [INF] from='client.? 172.21.15.44:0/2821758756' entity='client.admin' 2024-09-17T07:16:27.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:16:26.454101+0000 mgr.y (mgr.14144) 320 : cluster [DBG] pgmap v291: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:27.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:16:26.454101+0000 mgr.y (mgr.14144) 320 : cluster [DBG] pgmap v291: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:27.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:27 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:16:26.454101+0000 mgr.y (mgr.14144) 320 : cluster [DBG] pgmap v291: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:28.211 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-09-17T07:16:28.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:28 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:16:27.521901+0000 mon.a (mon.0) 610 : audit [INF] from='client.? ' entity='client.admin' 2024-09-17T07:16:28.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:28 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:16:27.525622+0000 mon.a (mon.0) 611 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:16:28.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:28 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:16:27.533118+0000 mon.a (mon.0) 612 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:16:28.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:28 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:16:27.521901+0000 mon.a (mon.0) 610 : audit [INF] from='client.? ' entity='client.admin' 2024-09-17T07:16:28.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:28 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:16:27.525622+0000 mon.a (mon.0) 611 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:16:28.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:28 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:16:27.533118+0000 mon.a (mon.0) 612 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:16:28.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:28 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:16:27.521901+0000 mon.a (mon.0) 610 : audit [INF] from='client.? ' entity='client.admin' 2024-09-17T07:16:28.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:28 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:16:27.525622+0000 mon.a (mon.0) 611 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:16:28.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:28 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:16:27.533118+0000 mon.a (mon.0) 612 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:16:29.772 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:16:28.454770+0000 mgr.y (mgr.14144) 321 : cluster [DBG] pgmap v292: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:29.772 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:16:28.454770+0000 mgr.y (mgr.14144) 321 : cluster [DBG] pgmap v292: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:29.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:16:28.454770+0000 mgr.y (mgr.14144) 321 : cluster [DBG] pgmap v292: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:30.216 INFO:teuthology.orchestra.run.smithi044.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T07:16:30.988 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-17T07:16:30.998 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi044.front.sepia.ceph.com 2024-09-17T07:16:30.998 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-09-17T07:16:31.218 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:16:30.211929+0000 mgr.y (mgr.14144) 322 : audit [DBG] from='client.24799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:16:31.218 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:16:30.212471+0000 mgr.y (mgr.14144) 323 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T07:16:31.218 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:16:30.216053+0000 mon.a (mon.0) 613 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:16:31.218 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:16:30.216563+0000 mon.a (mon.0) 614 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:16:31.218 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:16:30.221265+0000 mon.a (mon.0) 615 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:16:31.218 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:16:30.224740+0000 mgr.y (mgr.14144) 324 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T07:16:31.218 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:16:30.455238+0000 mgr.y (mgr.14144) 325 : cluster [DBG] pgmap v293: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:31.219 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:16:30.211929+0000 mgr.y (mgr.14144) 322 : audit [DBG] from='client.24799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:16:31.219 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:16:30.212471+0000 mgr.y (mgr.14144) 323 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T07:16:31.219 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:16:30.216053+0000 mon.a (mon.0) 613 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:16:31.219 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:16:30.216563+0000 mon.a (mon.0) 614 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:16:31.219 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:16:30.221265+0000 mon.a (mon.0) 615 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:16:31.219 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:16:30.224740+0000 mgr.y (mgr.14144) 324 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T07:16:31.219 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:16:30.455238+0000 mgr.y (mgr.14144) 325 : cluster [DBG] pgmap v293: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:31.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:16:30.211929+0000 mgr.y (mgr.14144) 322 : audit [DBG] from='client.24799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:16:31.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:16:30.212471+0000 mgr.y (mgr.14144) 323 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T07:16:31.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:16:30.216053+0000 mon.a (mon.0) 613 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:16:31.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:16:30.216563+0000 mon.a (mon.0) 614 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:16:31.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:16:30.221265+0000 mon.a (mon.0) 615 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:16:31.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:16:30.224740+0000 mgr.y (mgr.14144) 324 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T07:16:31.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:16:30.455238+0000 mgr.y (mgr.14144) 325 : cluster [DBG] pgmap v293: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:33.355 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-09-17T07:16:33.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:16:32.455976+0000 mgr.y (mgr.14144) 326 : cluster [DBG] pgmap v294: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:33.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:16:32.455976+0000 mgr.y (mgr.14144) 326 : cluster [DBG] pgmap v294: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:33.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:16:32.455976+0000 mgr.y (mgr.14144) 326 : cluster [DBG] pgmap v294: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:34.048 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST STATUS REFRESHED AGE PORTS VERSION IMAGE ID CONTAINER ID 2024-09-17T07:16:34.048 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.a smithi044 running (2m) 53s ago 2m *:9093 *:9094 0.20.0 0881eb8f169f 42aa02852f04 2024-09-17T07:16:34.048 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.a smithi179 running (2m) 116s ago 2m *:3000 6.7.4 557c83e11646 2436fcd841b8 2024-09-17T07:16:34.048 INFO:teuthology.orchestra.run.smithi044.stdout:iscsi.foo.smithi044.elmtfk smithi044 error 53s ago 2m - 2024-09-17T07:16:34.048 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.x smithi179 running (6m) 116s ago 6m *:8443 16.2.0 24ecd6d5f14c 1260ac34e181 2024-09-17T07:16:34.049 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.y smithi044 running (8m) 53s ago 8m *:9283 16.2.0 24ecd6d5f14c 1b5bd0a0cb11 2024-09-17T07:16:34.049 INFO:teuthology.orchestra.run.smithi044.stdout:mon.a smithi044 running (8m) 53s ago 8m - 16.2.0 24ecd6d5f14c 7905efa25a24 2024-09-17T07:16:34.049 INFO:teuthology.orchestra.run.smithi044.stdout:mon.b smithi179 running (7m) 116s ago 7m - 16.2.0 24ecd6d5f14c 198f81fe2641 2024-09-17T07:16:34.049 INFO:teuthology.orchestra.run.smithi044.stdout:mon.c smithi044 running (7m) 53s ago 7m - 16.2.0 24ecd6d5f14c e0d29cfe87a2 2024-09-17T07:16:34.049 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.a smithi044 running (3m) 53s ago 3m *:9100 0.18.1 e5a616e4b9cf d7284955755c 2024-09-17T07:16:34.049 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.b smithi179 running (3m) 116s ago 3m *:9100 0.18.1 e5a616e4b9cf 2106bcb6f0e8 2024-09-17T07:16:34.049 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (6m) 53s ago 6m - 16.2.0 24ecd6d5f14c 4517de6029ac 2024-09-17T07:16:34.049 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (6m) 53s ago 6m - 16.2.0 24ecd6d5f14c 2d5122cbeff9 2024-09-17T07:16:34.049 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (6m) 53s ago 6m - 16.2.0 24ecd6d5f14c 7fd2f30be874 2024-09-17T07:16:34.049 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi044 running (5m) 53s ago 5m - 16.2.0 24ecd6d5f14c 2b7d623395a7 2024-09-17T07:16:34.049 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi179 running (5m) 116s ago 5m - 16.2.0 24ecd6d5f14c 911e6f47c822 2024-09-17T07:16:34.049 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi179 running (5m) 116s ago 5m - 16.2.0 24ecd6d5f14c e56060742dff 2024-09-17T07:16:34.049 INFO:teuthology.orchestra.run.smithi044.stdout:osd.6 smithi179 running (4m) 116s ago 4m - 16.2.0 24ecd6d5f14c 7a5c30b68cbc 2024-09-17T07:16:34.050 INFO:teuthology.orchestra.run.smithi044.stdout:osd.7 smithi179 running (4m) 116s ago 4m - 16.2.0 24ecd6d5f14c b0d186fbf8a6 2024-09-17T07:16:34.050 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.a smithi179 running (2m) 116s ago 3m *:9095 2.18.1 de242295e225 3b50713c0562 2024-09-17T07:16:34.050 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi044.nhyhnh smithi044 running (2m) 53s ago 2m *:8000 16.2.0 24ecd6d5f14c 9b455479614f 2024-09-17T07:16:34.050 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi179.tqlzyy smithi179 running (2m) 116s ago 2m *:8000 16.2.0 24ecd6d5f14c c8d1693241ce 2024-09-17T07:16:34.050 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi044.vnufiz smithi044 running (2m) 53s ago 2m *:80 16.2.0 24ecd6d5f14c 6eae2bff5ec8 2024-09-17T07:16:34.050 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi179.ecjhll smithi179 running (2m) 116s ago 2m *:80 16.2.0 24ecd6d5f14c 253109ee0394 2024-09-17T07:16:34.429 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:16:34.429 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-09-17T07:16:34.429 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 3 2024-09-17T07:16:34.429 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:16:34.429 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-09-17T07:16:34.429 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 2 2024-09-17T07:16:34.429 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:16:34.429 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-09-17T07:16:34.429 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2024-09-17T07:16:34.430 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:16:34.430 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": {}, 2024-09-17T07:16:34.430 INFO:teuthology.orchestra.run.smithi044.stdout: "rgw": { 2024-09-17T07:16:34.430 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2024-09-17T07:16:34.430 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:16:34.430 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-09-17T07:16:34.430 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 17 2024-09-17T07:16:34.430 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-09-17T07:16:34.430 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:16:34.785 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:16:34.785 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9", 2024-09-17T07:16:34.786 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-09-17T07:16:34.786 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [], 2024-09-17T07:16:34.786 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "", 2024-09-17T07:16:34.786 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 image" 2024-09-17T07:16:34.786 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:16:34.786 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:16:33.338923+0000 mgr.y (mgr.14144) 327 : audit [DBG] from='client.14886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:16:34.786 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:16:34.428888+0000 mon.c (mon.1) 28 : audit [DBG] from='client.? 172.21.15.44:0/2317424955' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:16:34.786 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:16:33.338923+0000 mgr.y (mgr.14144) 327 : audit [DBG] from='client.14886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:16:34.787 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:16:34.428888+0000 mon.c (mon.1) 28 : audit [DBG] from='client.? 172.21.15.44:0/2317424955' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:16:34.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:16:33.338923+0000 mgr.y (mgr.14144) 327 : audit [DBG] from='client.14886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:16:34.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:16:34.428888+0000 mon.c (mon.1) 28 : audit [DBG] from='client.? 172.21.15.44:0/2317424955' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:16:35.166 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-09-17T07:16:35.166 INFO:teuthology.orchestra.run.smithi044.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-17T07:16:35.166 INFO:teuthology.orchestra.run.smithi044.stdout: daemon iscsi.foo.smithi044.elmtfk on smithi044 is in error state 2024-09-17T07:16:35.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:16:33.691576+0000 mgr.y (mgr.14144) 328 : audit [DBG] from='client.14892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:16:35.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:16:34.038992+0000 mgr.y (mgr.14144) 329 : audit [DBG] from='client.24817 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:16:35.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:16:34.456408+0000 mgr.y (mgr.14144) 330 : cluster [DBG] pgmap v295: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:35.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:16:35.166127+0000 mon.a (mon.0) 616 : audit [DBG] from='client.? 172.21.15.44:0/2031639328' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:16:35.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:16:33.691576+0000 mgr.y (mgr.14144) 328 : audit [DBG] from='client.14892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:16:35.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:16:34.038992+0000 mgr.y (mgr.14144) 329 : audit [DBG] from='client.24817 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:16:35.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:16:34.456408+0000 mgr.y (mgr.14144) 330 : cluster [DBG] pgmap v295: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:35.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:16:35.166127+0000 mon.a (mon.0) 616 : audit [DBG] from='client.? 172.21.15.44:0/2031639328' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:16:35.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:35 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:16:33.691576+0000 mgr.y (mgr.14144) 328 : audit [DBG] from='client.14892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:16:35.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:35 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:16:34.038992+0000 mgr.y (mgr.14144) 329 : audit [DBG] from='client.24817 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:16:35.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:35 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:16:34.456408+0000 mgr.y (mgr.14144) 330 : cluster [DBG] pgmap v295: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:35.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:35 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:16:35.166127+0000 mon.a (mon.0) 616 : audit [DBG] from='client.? 172.21.15.44:0/2031639328' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:16:36.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:16:34.785674+0000 mgr.y (mgr.14144) 331 : audit [DBG] from='client.14910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:16:36.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:16:34.785674+0000 mgr.y (mgr.14144) 331 : audit [DBG] from='client.14910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:16:36.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:16:34.785674+0000 mgr.y (mgr.14144) 331 : audit [DBG] from='client.14910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:16:37.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:16:36.457290+0000 mgr.y (mgr.14144) 332 : cluster [DBG] pgmap v296: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:37.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:16:36.457290+0000 mgr.y (mgr.14144) 332 : cluster [DBG] pgmap v296: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:37.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:37 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:16:36.457290+0000 mgr.y (mgr.14144) 332 : cluster [DBG] pgmap v296: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:39.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:16:38.457800+0000 mgr.y (mgr.14144) 333 : cluster [DBG] pgmap v297: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:39.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:16:38.457800+0000 mgr.y (mgr.14144) 333 : cluster [DBG] pgmap v297: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:39.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:16:38.457800+0000 mgr.y (mgr.14144) 333 : cluster [DBG] pgmap v297: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:41.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:16:40.458225+0000 mgr.y (mgr.14144) 334 : cluster [DBG] pgmap v298: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:41.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:16:40.458225+0000 mgr.y (mgr.14144) 334 : cluster [DBG] pgmap v298: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:41.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:40 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:16:40.458225+0000 mgr.y (mgr.14144) 334 : cluster [DBG] pgmap v298: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:43.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:43 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:16:42.458869+0000 mgr.y (mgr.14144) 335 : cluster [DBG] pgmap v299: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:43.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:43 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:16:42.458869+0000 mgr.y (mgr.14144) 335 : cluster [DBG] pgmap v299: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:43.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:43 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:16:42.458869+0000 mgr.y (mgr.14144) 335 : cluster [DBG] pgmap v299: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:45.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:16:44.459283+0000 mgr.y (mgr.14144) 336 : cluster [DBG] pgmap v300: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:45.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:16:44.459283+0000 mgr.y (mgr.14144) 336 : cluster [DBG] pgmap v300: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:45.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:16:44.459283+0000 mgr.y (mgr.14144) 336 : cluster [DBG] pgmap v300: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:47.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:16:46.459930+0000 mgr.y (mgr.14144) 337 : cluster [DBG] pgmap v301: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:47.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:16:46.459930+0000 mgr.y (mgr.14144) 337 : cluster [DBG] pgmap v301: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:47.863 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:16:46.459930+0000 mgr.y (mgr.14144) 337 : cluster [DBG] pgmap v301: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:48.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:16:47.877879+0000 mon.a (mon.0) 617 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:16:48.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:16:47.878102+0000 mon.a (mon.0) 618 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:16:48.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:16:47.877879+0000 mon.a (mon.0) 617 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:16:48.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:16:47.878102+0000 mon.a (mon.0) 618 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:16:48.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:16:47.877879+0000 mon.a (mon.0) 617 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:16:48.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:16:47.878102+0000 mon.a (mon.0) 618 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:16:49.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:16:48.460404+0000 mgr.y (mgr.14144) 338 : cluster [DBG] pgmap v302: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:49.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:16:48.460404+0000 mgr.y (mgr.14144) 338 : cluster [DBG] pgmap v302: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:49.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:16:48.460404+0000 mgr.y (mgr.14144) 338 : cluster [DBG] pgmap v302: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:51.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:16:50.460812+0000 mgr.y (mgr.14144) 339 : cluster [DBG] pgmap v303: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:51.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:16:50.460812+0000 mgr.y (mgr.14144) 339 : cluster [DBG] pgmap v303: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:51.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:16:50.460812+0000 mgr.y (mgr.14144) 339 : cluster [DBG] pgmap v303: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:53.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:53 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:16:52.461451+0000 mgr.y (mgr.14144) 340 : cluster [DBG] pgmap v304: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:53.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:53 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:16:52.461451+0000 mgr.y (mgr.14144) 340 : cluster [DBG] pgmap v304: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:53.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:16:52.461451+0000 mgr.y (mgr.14144) 340 : cluster [DBG] pgmap v304: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:55.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:16:54.461890+0000 mgr.y (mgr.14144) 341 : cluster [DBG] pgmap v305: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:55.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:16:54.461890+0000 mgr.y (mgr.14144) 341 : cluster [DBG] pgmap v305: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:55.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:16:54.461890+0000 mgr.y (mgr.14144) 341 : cluster [DBG] pgmap v305: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:57.521 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:16:56.462540+0000 mgr.y (mgr.14144) 342 : cluster [DBG] pgmap v306: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:57.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:16:56.462540+0000 mgr.y (mgr.14144) 342 : cluster [DBG] pgmap v306: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:57.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:16:56.462540+0000 mgr.y (mgr.14144) 342 : cluster [DBG] pgmap v306: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:59.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:16:59 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:16:58.463056+0000 mgr.y (mgr.14144) 343 : cluster [DBG] pgmap v307: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:59.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:16:59 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:16:58.463056+0000 mgr.y (mgr.14144) 343 : cluster [DBG] pgmap v307: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:16:59.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:16:59 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:16:58.463056+0000 mgr.y (mgr.14144) 343 : cluster [DBG] pgmap v307: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:01.078 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:00.463470+0000 mgr.y (mgr.14144) 344 : cluster [DBG] pgmap v308: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:01.078 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:00.463470+0000 mgr.y (mgr.14144) 344 : cluster [DBG] pgmap v308: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:01.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:00 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:00.463470+0000 mgr.y (mgr.14144) 344 : cluster [DBG] pgmap v308: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:03.817 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:03 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:02.464363+0000 mgr.y (mgr.14144) 345 : cluster [DBG] pgmap v309: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:03.817 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:03 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:03.169654+0000 mon.a (mon.0) 619 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:17:03.817 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:03 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:03.171262+0000 mon.a (mon.0) 620 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:17:03.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:03 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:02.464363+0000 mgr.y (mgr.14144) 345 : cluster [DBG] pgmap v309: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:03.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:03 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:03.169654+0000 mon.a (mon.0) 619 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:17:03.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:03 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:03.171262+0000 mon.a (mon.0) 620 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:17:03.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:03 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:02.464363+0000 mgr.y (mgr.14144) 345 : cluster [DBG] pgmap v309: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:03.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:03 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:03.169654+0000 mon.a (mon.0) 619 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:17:03.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:03 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:03.171262+0000 mon.a (mon.0) 620 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:17:04.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:04 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:17:03.170259+0000 mgr.y (mgr.14144) 346 : cephadm [INF] Upgrade: Target is version 18.2.4-855-g19e2a2be (unknown) 2024-09-17T07:17:04.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:04 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:17:03.170361+0000 mgr.y (mgr.14144) 347 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, digests ['quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9'] 2024-09-17T07:17:04.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:04 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:17:03.172192+0000 mgr.y (mgr.14144) 348 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2024-09-17T07:17:04.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:04 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:17:03.377453+0000 mgr.y (mgr.14144) 349 : cephadm [INF] Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 on smithi179 2024-09-17T07:17:04.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:04 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:17:03.170259+0000 mgr.y (mgr.14144) 346 : cephadm [INF] Upgrade: Target is version 18.2.4-855-g19e2a2be (unknown) 2024-09-17T07:17:04.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:04 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:17:03.170361+0000 mgr.y (mgr.14144) 347 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, digests ['quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9'] 2024-09-17T07:17:04.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:04 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:17:03.172192+0000 mgr.y (mgr.14144) 348 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2024-09-17T07:17:04.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:04 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:17:03.377453+0000 mgr.y (mgr.14144) 349 : cephadm [INF] Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 on smithi179 2024-09-17T07:17:04.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:04 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:17:03.170259+0000 mgr.y (mgr.14144) 346 : cephadm [INF] Upgrade: Target is version 18.2.4-855-g19e2a2be (unknown) 2024-09-17T07:17:04.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:04 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:17:03.170361+0000 mgr.y (mgr.14144) 347 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, digests ['quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9'] 2024-09-17T07:17:04.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:04 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:17:03.172192+0000 mgr.y (mgr.14144) 348 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2024-09-17T07:17:04.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:04 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:17:03.377453+0000 mgr.y (mgr.14144) 349 : cephadm [INF] Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 on smithi179 2024-09-17T07:17:05.503 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-09-17T07:17:05.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:04.465031+0000 mgr.y (mgr.14144) 350 : cluster [DBG] pgmap v310: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:05.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:04.465031+0000 mgr.y (mgr.14144) 350 : cluster [DBG] pgmap v310: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:05.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:05 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:04.465031+0000 mgr.y (mgr.14144) 350 : cluster [DBG] pgmap v310: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:06.106 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST STATUS REFRESHED AGE PORTS VERSION IMAGE ID CONTAINER ID 2024-09-17T07:17:06.106 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.a smithi044 running (3m) 85s ago 3m *:9093 *:9094 0.20.0 0881eb8f169f 42aa02852f04 2024-09-17T07:17:06.107 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.a smithi179 running (3m) 2m ago 3m *:3000 6.7.4 557c83e11646 2436fcd841b8 2024-09-17T07:17:06.107 INFO:teuthology.orchestra.run.smithi044.stdout:iscsi.foo.smithi044.elmtfk smithi044 error 85s ago 2m - 2024-09-17T07:17:06.107 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.x smithi179 running (7m) 2m ago 7m *:8443 16.2.0 24ecd6d5f14c 1260ac34e181 2024-09-17T07:17:06.107 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.y smithi044 running (8m) 85s ago 8m *:9283 16.2.0 24ecd6d5f14c 1b5bd0a0cb11 2024-09-17T07:17:06.107 INFO:teuthology.orchestra.run.smithi044.stdout:mon.a smithi044 running (8m) 85s ago 9m - 16.2.0 24ecd6d5f14c 7905efa25a24 2024-09-17T07:17:06.107 INFO:teuthology.orchestra.run.smithi044.stdout:mon.b smithi179 running (7m) 2m ago 7m - 16.2.0 24ecd6d5f14c 198f81fe2641 2024-09-17T07:17:06.107 INFO:teuthology.orchestra.run.smithi044.stdout:mon.c smithi044 running (7m) 85s ago 7m - 16.2.0 24ecd6d5f14c e0d29cfe87a2 2024-09-17T07:17:06.107 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.a smithi044 running (3m) 85s ago 3m *:9100 0.18.1 e5a616e4b9cf d7284955755c 2024-09-17T07:17:06.107 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.b smithi179 running (4m) 2m ago 4m *:9100 0.18.1 e5a616e4b9cf 2106bcb6f0e8 2024-09-17T07:17:06.107 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (7m) 85s ago 7m - 16.2.0 24ecd6d5f14c 4517de6029ac 2024-09-17T07:17:06.107 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (6m) 85s ago 6m - 16.2.0 24ecd6d5f14c 2d5122cbeff9 2024-09-17T07:17:06.107 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (6m) 85s ago 6m - 16.2.0 24ecd6d5f14c 7fd2f30be874 2024-09-17T07:17:06.107 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi044 running (6m) 85s ago 6m - 16.2.0 24ecd6d5f14c 2b7d623395a7 2024-09-17T07:17:06.108 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi179 running (6m) 2m ago 6m - 16.2.0 24ecd6d5f14c 911e6f47c822 2024-09-17T07:17:06.108 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi179 running (5m) 2m ago 5m - 16.2.0 24ecd6d5f14c e56060742dff 2024-09-17T07:17:06.108 INFO:teuthology.orchestra.run.smithi044.stdout:osd.6 smithi179 running (5m) 2m ago 5m - 16.2.0 24ecd6d5f14c 7a5c30b68cbc 2024-09-17T07:17:06.108 INFO:teuthology.orchestra.run.smithi044.stdout:osd.7 smithi179 running (5m) 2m ago 5m - 16.2.0 24ecd6d5f14c b0d186fbf8a6 2024-09-17T07:17:06.108 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.a smithi179 running (2m) 2m ago 4m *:9095 2.18.1 de242295e225 3b50713c0562 2024-09-17T07:17:06.108 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi044.nhyhnh smithi044 running (2m) 85s ago 2m *:8000 16.2.0 24ecd6d5f14c 9b455479614f 2024-09-17T07:17:06.108 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi179.tqlzyy smithi179 running (2m) 2m ago 2m *:8000 16.2.0 24ecd6d5f14c c8d1693241ce 2024-09-17T07:17:06.108 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi044.vnufiz smithi044 running (2m) 85s ago 2m *:80 16.2.0 24ecd6d5f14c 6eae2bff5ec8 2024-09-17T07:17:06.108 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi179.ecjhll smithi179 running (2m) 2m ago 2m *:80 16.2.0 24ecd6d5f14c 253109ee0394 2024-09-17T07:17:06.434 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:17:06.434 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-09-17T07:17:06.434 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 3 2024-09-17T07:17:06.434 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:17:06.435 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-09-17T07:17:06.435 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 2 2024-09-17T07:17:06.435 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:17:06.435 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-09-17T07:17:06.435 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2024-09-17T07:17:06.435 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:17:06.435 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": {}, 2024-09-17T07:17:06.435 INFO:teuthology.orchestra.run.smithi044.stdout: "rgw": { 2024-09-17T07:17:06.435 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2024-09-17T07:17:06.436 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:17:06.436 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-09-17T07:17:06.436 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 17 2024-09-17T07:17:06.436 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-09-17T07:17:06.436 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:17:06.743 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:17:06.744 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9", 2024-09-17T07:17:06.744 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-09-17T07:17:06.744 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [], 2024-09-17T07:17:06.744 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "0/23 ceph daemons upgraded", 2024-09-17T07:17:06.744 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 image on host smithi179" 2024-09-17T07:17:06.744 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:17:06.744 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:05.490718+0000 mgr.y (mgr.14144) 351 : audit [DBG] from='client.24841 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:17:06.744 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:06.434238+0000 mon.a (mon.0) 621 : audit [DBG] from='client.? 172.21.15.44:0/1725743199' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:17:06.745 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:05.490718+0000 mgr.y (mgr.14144) 351 : audit [DBG] from='client.24841 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:17:06.745 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:06.434238+0000 mon.a (mon.0) 621 : audit [DBG] from='client.? 172.21.15.44:0/1725743199' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:17:06.863 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:05.490718+0000 mgr.y (mgr.14144) 351 : audit [DBG] from='client.24841 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:17:06.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:06.434238+0000 mon.a (mon.0) 621 : audit [DBG] from='client.? 172.21.15.44:0/1725743199' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:17:07.079 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-09-17T07:17:07.079 INFO:teuthology.orchestra.run.smithi044.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-17T07:17:07.079 INFO:teuthology.orchestra.run.smithi044.stdout: daemon iscsi.foo.smithi044.elmtfk on smithi044 is in error state 2024-09-17T07:17:07.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:05.801448+0000 mgr.y (mgr.14144) 352 : audit [DBG] from='client.14928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:17:07.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:06.099097+0000 mgr.y (mgr.14144) 353 : audit [DBG] from='client.24853 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:17:07.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:06.465669+0000 mgr.y (mgr.14144) 354 : cluster [DBG] pgmap v311: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:07.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:07.079760+0000 mon.c (mon.1) 29 : audit [DBG] from='client.? 172.21.15.44:0/1243920850' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:17:07.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:05.801448+0000 mgr.y (mgr.14144) 352 : audit [DBG] from='client.14928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:17:07.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:06.099097+0000 mgr.y (mgr.14144) 353 : audit [DBG] from='client.24853 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:17:07.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:06.465669+0000 mgr.y (mgr.14144) 354 : cluster [DBG] pgmap v311: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:07.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:07.079760+0000 mon.c (mon.1) 29 : audit [DBG] from='client.? 172.21.15.44:0/1243920850' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:17:07.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:05.801448+0000 mgr.y (mgr.14144) 352 : audit [DBG] from='client.14928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:17:07.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:06.099097+0000 mgr.y (mgr.14144) 353 : audit [DBG] from='client.24853 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:17:07.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:06.465669+0000 mgr.y (mgr.14144) 354 : cluster [DBG] pgmap v311: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:07.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:07.079760+0000 mon.c (mon.1) 29 : audit [DBG] from='client.? 172.21.15.44:0/1243920850' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:17:08.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:06.743618+0000 mgr.y (mgr.14144) 355 : audit [DBG] from='client.24865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:17:08.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:06.743618+0000 mgr.y (mgr.14144) 355 : audit [DBG] from='client.24865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:17:08.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:06.743618+0000 mgr.y (mgr.14144) 355 : audit [DBG] from='client.24865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:17:09.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:08.466254+0000 mgr.y (mgr.14144) 356 : cluster [DBG] pgmap v312: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:09.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:08.466254+0000 mgr.y (mgr.14144) 356 : cluster [DBG] pgmap v312: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:09.863 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:08.466254+0000 mgr.y (mgr.14144) 356 : cluster [DBG] pgmap v312: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:11.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:10.466722+0000 mgr.y (mgr.14144) 357 : cluster [DBG] pgmap v313: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:11.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:10.466722+0000 mgr.y (mgr.14144) 357 : cluster [DBG] pgmap v313: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:11.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:10.466722+0000 mgr.y (mgr.14144) 357 : cluster [DBG] pgmap v313: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:13.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:12.467493+0000 mgr.y (mgr.14144) 358 : cluster [DBG] pgmap v314: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:13.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:12.467493+0000 mgr.y (mgr.14144) 358 : cluster [DBG] pgmap v314: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:13.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:13 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:12.467493+0000 mgr.y (mgr.14144) 358 : cluster [DBG] pgmap v314: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:15.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:14.467990+0000 mgr.y (mgr.14144) 359 : cluster [DBG] pgmap v315: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:15.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:14.467990+0000 mgr.y (mgr.14144) 359 : cluster [DBG] pgmap v315: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:15.863 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:14.467990+0000 mgr.y (mgr.14144) 359 : cluster [DBG] pgmap v315: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:17.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:17 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:16.468740+0000 mgr.y (mgr.14144) 360 : cluster [DBG] pgmap v316: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:17.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:17 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:16.468740+0000 mgr.y (mgr.14144) 360 : cluster [DBG] pgmap v316: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:17.863 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:16.468740+0000 mgr.y (mgr.14144) 360 : cluster [DBG] pgmap v316: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:19.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:18.469387+0000 mgr.y (mgr.14144) 361 : cluster [DBG] pgmap v317: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:19.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:18.469387+0000 mgr.y (mgr.14144) 361 : cluster [DBG] pgmap v317: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:19.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:19 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:18.469387+0000 mgr.y (mgr.14144) 361 : cluster [DBG] pgmap v317: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:21.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:20.470085+0000 mgr.y (mgr.14144) 362 : cluster [DBG] pgmap v318: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:21.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:20.470085+0000 mgr.y (mgr.14144) 362 : cluster [DBG] pgmap v318: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:21.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:20.470085+0000 mgr.y (mgr.14144) 362 : cluster [DBG] pgmap v318: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:23.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:23 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:22.470862+0000 mgr.y (mgr.14144) 363 : cluster [DBG] pgmap v319: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:23.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:23 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:22.470862+0000 mgr.y (mgr.14144) 363 : cluster [DBG] pgmap v319: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:23.863 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:22.470862+0000 mgr.y (mgr.14144) 363 : cluster [DBG] pgmap v319: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:25.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:24.471553+0000 mgr.y (mgr.14144) 364 : cluster [DBG] pgmap v320: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:25.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:24.471553+0000 mgr.y (mgr.14144) 364 : cluster [DBG] pgmap v320: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:25.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:25 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:24.471553+0000 mgr.y (mgr.14144) 364 : cluster [DBG] pgmap v320: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:27.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:26.472291+0000 mgr.y (mgr.14144) 365 : cluster [DBG] pgmap v321: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:27.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:26.472291+0000 mgr.y (mgr.14144) 365 : cluster [DBG] pgmap v321: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:27.863 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:27 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:26.472291+0000 mgr.y (mgr.14144) 365 : cluster [DBG] pgmap v321: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:29.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:28.473037+0000 mgr.y (mgr.14144) 366 : cluster [DBG] pgmap v322: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:29.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:28.473037+0000 mgr.y (mgr.14144) 366 : cluster [DBG] pgmap v322: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:29.863 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:28.473037+0000 mgr.y (mgr.14144) 366 : cluster [DBG] pgmap v322: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:31.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:30.473502+0000 mgr.y (mgr.14144) 367 : cluster [DBG] pgmap v323: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:31.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:30.473502+0000 mgr.y (mgr.14144) 367 : cluster [DBG] pgmap v323: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:31.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:30 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:30.473502+0000 mgr.y (mgr.14144) 367 : cluster [DBG] pgmap v323: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:33.815 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:32.474268+0000 mgr.y (mgr.14144) 368 : cluster [DBG] pgmap v324: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:33.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:32.474268+0000 mgr.y (mgr.14144) 368 : cluster [DBG] pgmap v324: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:33.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:32.474268+0000 mgr.y (mgr.14144) 368 : cluster [DBG] pgmap v324: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:35.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:34.474939+0000 mgr.y (mgr.14144) 369 : cluster [DBG] pgmap v325: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:35.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:34.474939+0000 mgr.y (mgr.14144) 369 : cluster [DBG] pgmap v325: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:35.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:35 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:34.474939+0000 mgr.y (mgr.14144) 369 : cluster [DBG] pgmap v325: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:37.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:17:36.006723+0000 mgr.y (mgr.14144) 370 : cephadm [INF] Upgrade: Updating mgr.x 2024-09-17T07:17:37.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:36.013416+0000 mon.a (mon.0) 622 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:17:37.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:36.014352+0000 mon.a (mon.0) 623 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T07:17:37.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:36.015258+0000 mon.a (mon.0) 624 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T07:17:37.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:36.016039+0000 mon.a (mon.0) 625 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:17:37.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:17:36.016871+0000 mgr.y (mgr.14144) 371 : cephadm [INF] Deploying daemon mgr.x on smithi179 2024-09-17T07:17:37.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:36.475793+0000 mgr.y (mgr.14144) 372 : cluster [DBG] pgmap v326: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:37.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:17:36.006723+0000 mgr.y (mgr.14144) 370 : cephadm [INF] Upgrade: Updating mgr.x 2024-09-17T07:17:37.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:36.013416+0000 mon.a (mon.0) 622 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:17:37.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:36.014352+0000 mon.a (mon.0) 623 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T07:17:37.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:36.015258+0000 mon.a (mon.0) 624 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T07:17:37.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:36.016039+0000 mon.a (mon.0) 625 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:17:37.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:17:36.016871+0000 mgr.y (mgr.14144) 371 : cephadm [INF] Deploying daemon mgr.x on smithi179 2024-09-17T07:17:37.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:36.475793+0000 mgr.y (mgr.14144) 372 : cluster [DBG] pgmap v326: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:37.327 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:37 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:17:36.006723+0000 mgr.y (mgr.14144) 370 : cephadm [INF] Upgrade: Updating mgr.x 2024-09-17T07:17:37.327 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:37 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:36.013416+0000 mon.a (mon.0) 622 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:17:37.327 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:37 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:36.014352+0000 mon.a (mon.0) 623 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T07:17:37.327 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:37 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:36.015258+0000 mon.a (mon.0) 624 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T07:17:37.327 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:37 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:36.016039+0000 mon.a (mon.0) 625 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:17:37.327 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:37 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:17:36.016871+0000 mgr.y (mgr.14144) 371 : cephadm [INF] Deploying daemon mgr.x on smithi179 2024-09-17T07:17:37.327 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:37 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:36.475793+0000 mgr.y (mgr.14144) 372 : cluster [DBG] pgmap v326: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:37.416 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-09-17T07:17:37.614 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:37 smithi179 systemd[1]: Stopping Ceph mgr.x for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:17:38.017 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST STATUS REFRESHED AGE PORTS VERSION IMAGE ID CONTAINER ID 2024-09-17T07:17:38.017 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.a smithi044 running (3m) 117s ago 4m *:9093 *:9094 0.20.0 0881eb8f169f 42aa02852f04 2024-09-17T07:17:38.017 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.a smithi179 running (3m) 3m ago 3m *:3000 6.7.4 557c83e11646 2436fcd841b8 2024-09-17T07:17:38.017 INFO:teuthology.orchestra.run.smithi044.stdout:iscsi.foo.smithi044.elmtfk smithi044 error 117s ago 3m - 2024-09-17T07:17:38.017 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.x smithi179 running (8m) 3m ago 8m *:8443 16.2.0 24ecd6d5f14c 1260ac34e181 2024-09-17T07:17:38.017 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.y smithi044 running (9m) 117s ago 9m *:9283 16.2.0 24ecd6d5f14c 1b5bd0a0cb11 2024-09-17T07:17:38.018 INFO:teuthology.orchestra.run.smithi044.stdout:mon.a smithi044 running (9m) 117s ago 9m - 16.2.0 24ecd6d5f14c 7905efa25a24 2024-09-17T07:17:38.018 INFO:teuthology.orchestra.run.smithi044.stdout:mon.b smithi179 running (8m) 3m ago 8m - 16.2.0 24ecd6d5f14c 198f81fe2641 2024-09-17T07:17:38.018 INFO:teuthology.orchestra.run.smithi044.stdout:mon.c smithi044 running (8m) 117s ago 8m - 16.2.0 24ecd6d5f14c e0d29cfe87a2 2024-09-17T07:17:38.018 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.a smithi044 running (4m) 117s ago 4m *:9100 0.18.1 e5a616e4b9cf d7284955755c 2024-09-17T07:17:38.018 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.b smithi179 running (4m) 3m ago 4m *:9100 0.18.1 e5a616e4b9cf 2106bcb6f0e8 2024-09-17T07:17:38.018 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (7m) 117s ago 7m - 16.2.0 24ecd6d5f14c 4517de6029ac 2024-09-17T07:17:38.018 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (7m) 117s ago 7m - 16.2.0 24ecd6d5f14c 2d5122cbeff9 2024-09-17T07:17:38.018 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (7m) 117s ago 7m - 16.2.0 24ecd6d5f14c 7fd2f30be874 2024-09-17T07:17:38.018 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi044 running (6m) 117s ago 6m - 16.2.0 24ecd6d5f14c 2b7d623395a7 2024-09-17T07:17:38.018 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi179 running (6m) 3m ago 6m - 16.2.0 24ecd6d5f14c 911e6f47c822 2024-09-17T07:17:38.018 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi179 running (6m) 3m ago 6m - 16.2.0 24ecd6d5f14c e56060742dff 2024-09-17T07:17:38.018 INFO:teuthology.orchestra.run.smithi044.stdout:osd.6 smithi179 running (5m) 3m ago 5m - 16.2.0 24ecd6d5f14c 7a5c30b68cbc 2024-09-17T07:17:38.018 INFO:teuthology.orchestra.run.smithi044.stdout:osd.7 smithi179 running (5m) 3m ago 5m - 16.2.0 24ecd6d5f14c b0d186fbf8a6 2024-09-17T07:17:38.019 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.a smithi179 running (3m) 3m ago 5m *:9095 2.18.1 de242295e225 3b50713c0562 2024-09-17T07:17:38.019 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi044.nhyhnh smithi044 running (3m) 117s ago 3m *:8000 16.2.0 24ecd6d5f14c 9b455479614f 2024-09-17T07:17:38.019 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi179.tqlzyy smithi179 running (3m) 3m ago 3m *:8000 16.2.0 24ecd6d5f14c c8d1693241ce 2024-09-17T07:17:38.019 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi044.vnufiz smithi044 running (3m) 117s ago 3m *:80 16.2.0 24ecd6d5f14c 6eae2bff5ec8 2024-09-17T07:17:38.019 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi179.ecjhll smithi179 running (3m) 3m ago 3m *:80 16.2.0 24ecd6d5f14c 253109ee0394 2024-09-17T07:17:38.181 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:38 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:37.403423+0000 mgr.y (mgr.14144) 373 : audit [DBG] from='client.14958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:17:38.182 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:37 smithi179 podman[40839]: 2024-09-17 07:17:37.909874088 +0000 UTC m=+0.350542118 container died 1260ac34e181076e6ff0914c6a7020cde4d19cfbc2bb6903d4b4520f3bc76ce9 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x, GIT_BRANCH=HEAD, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, CEPH_POINT_RELEASE=-16.2.0, ceph=True, maintainer=Dimitri Savineau , GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.build-date=20201204, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image) 2024-09-17T07:17:38.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:37.403423+0000 mgr.y (mgr.14144) 373 : audit [DBG] from='client.14958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:17:38.324 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:37.403423+0000 mgr.y (mgr.14144) 373 : audit [DBG] from='client.14958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:17:38.349 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:17:38.349 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-09-17T07:17:38.349 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 3 2024-09-17T07:17:38.349 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:17:38.349 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-09-17T07:17:38.349 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 2 2024-09-17T07:17:38.349 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:17:38.349 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-09-17T07:17:38.349 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2024-09-17T07:17:38.349 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:17:38.350 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": {}, 2024-09-17T07:17:38.350 INFO:teuthology.orchestra.run.smithi044.stdout: "rgw": { 2024-09-17T07:17:38.350 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2024-09-17T07:17:38.350 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:17:38.350 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-09-17T07:17:38.350 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 17 2024-09-17T07:17:38.350 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-09-17T07:17:38.350 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:17:38.434 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:38 smithi179 podman[40839]: 2024-09-17 07:17:38.238980763 +0000 UTC m=+0.679648823 container cleanup 1260ac34e181076e6ff0914c6a7020cde4d19cfbc2bb6903d4b4520f3bc76ce9 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x, org.label-schema.name=CentOS Base Image, RELEASE=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Dimitri Savineau , CEPH_POINT_RELEASE=-16.2.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, org.label-schema.build-date=20201204, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2) 2024-09-17T07:17:38.434 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:38 smithi179 podman[40839]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x 2024-09-17T07:17:38.434 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:38 smithi179 podman[40851]: 2024-09-17 07:17:38.431736709 +0000 UTC m=+0.520095973 container remove 1260ac34e181076e6ff0914c6a7020cde4d19cfbc2bb6903d4b4520f3bc76ce9 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x, org.label-schema.build-date=20201204, RELEASE=HEAD, maintainer=Dimitri Savineau , GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-16.2.0, GIT_BRANCH=HEAD) 2024-09-17T07:17:38.434 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:38 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-17T07:17:38.434 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:38 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-09-17T07:17:38.657 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:17:38.657 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9", 2024-09-17T07:17:38.657 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-09-17T07:17:38.657 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [], 2024-09-17T07:17:38.657 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "0/23 ceph daemons upgraded", 2024-09-17T07:17:38.658 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "Currently upgrading mgr daemons" 2024-09-17T07:17:38.658 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:17:38.791 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:38 smithi179 systemd[1]: Stopped Ceph mgr.x for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:17:38.791 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:38 smithi179 systemd[1]: Starting Ceph mgr.x for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:17:38.987 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-09-17T07:17:38.987 INFO:teuthology.orchestra.run.smithi044.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-17T07:17:38.987 INFO:teuthology.orchestra.run.smithi044.stdout: daemon iscsi.foo.smithi044.elmtfk on smithi044 is in error state 2024-09-17T07:17:39.069 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:37.709924+0000 mgr.y (mgr.14144) 374 : audit [DBG] from='client.24883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:17:39.069 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:38.009396+0000 mgr.y (mgr.14144) 375 : audit [DBG] from='client.24889 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:17:39.069 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:38.349106+0000 mon.c (mon.1) 30 : audit [DBG] from='client.? 172.21.15.44:0/176044143' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:17:39.069 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:38.476288+0000 mgr.y (mgr.14144) 376 : cluster [DBG] pgmap v327: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:39.069 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:38.987213+0000 mon.c (mon.1) 31 : audit [DBG] from='client.? 172.21.15.44:0/1666418441' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:17:39.070 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:38 smithi179 podman[40933]: 2024-09-17 07:17:38.791424733 +0000 UTC m=+0.097691878 container create c9b478bf98747d4662e64411af3650acc640bc350c0172e9e5693fd2b0622419 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-17T07:17:39.070 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:38 smithi179 podman[40933]: 2024-09-17 07:17:38.709530797 +0000 UTC m=+0.015797952 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:17:39.070 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:38 smithi179 podman[40933]: 2024-09-17 07:17:38.960209173 +0000 UTC m=+0.266476319 container init c9b478bf98747d4662e64411af3650acc640bc350c0172e9e5693fd2b0622419 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x, GIT_CLEAN=True, RELEASE=reef-19e2a2b, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-17T07:17:39.070 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:38 smithi179 podman[40933]: 2024-09-17 07:17:38.962931891 +0000 UTC m=+0.269199043 container start c9b478bf98747d4662e64411af3650acc640bc350c0172e9e5693fd2b0622419 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b) 2024-09-17T07:17:39.070 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:38 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:17:38.986+0000 7f1fedeac640 1 -- 172.21.15.179:0/692945174 <== mon.2 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b2871e2d00 con 0x55b2871b6800 2024-09-17T07:17:39.071 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:39 smithi179 bash[40933]: c9b478bf98747d4662e64411af3650acc640bc350c0172e9e5693fd2b0622419 2024-09-17T07:17:39.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:37.709924+0000 mgr.y (mgr.14144) 374 : audit [DBG] from='client.24883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:17:39.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:38.009396+0000 mgr.y (mgr.14144) 375 : audit [DBG] from='client.24889 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:17:39.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:38.349106+0000 mon.c (mon.1) 30 : audit [DBG] from='client.? 172.21.15.44:0/176044143' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:17:39.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:38.476288+0000 mgr.y (mgr.14144) 376 : cluster [DBG] pgmap v327: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:39.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:38.987213+0000 mon.c (mon.1) 31 : audit [DBG] from='client.? 172.21.15.44:0/1666418441' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:17:39.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:37.709924+0000 mgr.y (mgr.14144) 374 : audit [DBG] from='client.24883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:17:39.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:38.009396+0000 mgr.y (mgr.14144) 375 : audit [DBG] from='client.24889 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:17:39.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:38.349106+0000 mon.c (mon.1) 30 : audit [DBG] from='client.? 172.21.15.44:0/176044143' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:17:39.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:38.476288+0000 mgr.y (mgr.14144) 376 : cluster [DBG] pgmap v327: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:39.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:38.987213+0000 mon.c (mon.1) 31 : audit [DBG] from='client.? 172.21.15.44:0/1666418441' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:17:39.364 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:39 smithi179 systemd[1]: Started Ceph mgr.x for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:17:39.364 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:17:39.082+0000 7f1ff0715040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T07:17:39.364 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:17:39.142+0000 7f1ff0715040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T07:17:39.364 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:17:39.307+0000 7f1ff0715040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T07:17:39.679 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:17:39.385+0000 7f1ff0715040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T07:17:39.680 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:17:39.534+0000 7f1ff0715040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T07:17:39.680 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:17:39.678+0000 7f1ff0715040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T07:17:40.270 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:40 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:38.657039+0000 mgr.y (mgr.14144) 377 : audit [DBG] from='client.24901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:17:40.270 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:40 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:39.114480+0000 mon.a (mon.0) 626 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:17:40.270 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:40 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:39.115175+0000 mon.a (mon.0) 627 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:17:40.271 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:17:39.998+0000 7f1ff0715040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T07:17:40.271 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:40 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:17:40.059+0000 7f1ff0715040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T07:17:40.271 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:40 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:17:40.269+0000 7f1ff0715040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T07:17:40.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:38.657039+0000 mgr.y (mgr.14144) 377 : audit [DBG] from='client.24901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:17:40.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:39.114480+0000 mon.a (mon.0) 626 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:17:40.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:39.115175+0000 mon.a (mon.0) 627 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:17:40.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:38.657039+0000 mgr.y (mgr.14144) 377 : audit [DBG] from='client.24901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:17:40.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:39.114480+0000 mon.a (mon.0) 626 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:17:40.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:39.115175+0000 mon.a (mon.0) 627 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:17:41.454 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:41 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:40.476784+0000 mgr.y (mgr.14144) 378 : cluster [DBG] pgmap v328: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:41.455 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:41 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:17:41.232+0000 7f1ff0715040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T07:17:41.455 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:41 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:17:41.301+0000 7f1ff0715040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T07:17:41.574 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:41 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:40.476784+0000 mgr.y (mgr.14144) 378 : cluster [DBG] pgmap v328: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:41.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:41 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:40.476784+0000 mgr.y (mgr.14144) 378 : cluster [DBG] pgmap v328: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:42.153 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:17:42.055+0000 7f1ff0715040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T07:17:42.426 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:17:42.172+0000 7f1ff0715040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T07:17:42.426 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:17:42.318+0000 7f1ff0715040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T07:17:42.426 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:17:42.384+0000 7f1ff0715040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T07:17:42.845 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:17:42.534+0000 7f1ff0715040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T07:17:42.845 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:17:42.601+0000 7f1ff0715040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T07:17:42.845 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:17:42.673+0000 7f1ff0715040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T07:17:42.845 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:17:42.844+0000 7f1ff0715040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T07:17:43.106 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:17:42.911+0000 7f1ff0715040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T07:17:43.106 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:17:42.981+0000 7f1ff0715040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T07:17:43.106 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:43 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:17:43.065+0000 7f1ff0715040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T07:17:43.364 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:43 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:17:43.126+0000 7f1ff0715040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T07:17:43.364 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:43 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:17:43.185+0000 7f1ff0715040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T07:17:43.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:43 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:42.477534+0000 mgr.y (mgr.14144) 379 : cluster [DBG] pgmap v329: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:43.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:43 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:42.477534+0000 mgr.y (mgr.14144) 379 : cluster [DBG] pgmap v329: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:43.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:43 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:42.477534+0000 mgr.y (mgr.14144) 379 : cluster [DBG] pgmap v329: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:44.364 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:17:44.006+0000 7f1ff0715040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T07:17:44.804 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:43.543827+0000 mon.a (mon.0) 628 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:17:44.804 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:43.765465+0000 mon.a (mon.0) 629 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:17:44.804 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:43.769997+0000 mon.a (mon.0) 630 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:17:44.805 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:43.773409+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-17T07:17:44.805 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:43.778278+0000 mon.a (mon.0) 632 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-09-17T07:17:44.805 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:43.881348+0000 mon.a (mon.0) 633 : cluster [DBG] Standby manager daemon y started 2024-09-17T07:17:44.805 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:44.009432+0000 mon.a (mon.0) 634 : cluster [INF] Active manager daemon x restarted 2024-09-17T07:17:44.805 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:44.010279+0000 mon.a (mon.0) 635 : cluster [INF] Activating manager daemon x 2024-09-17T07:17:44.805 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:44.010391+0000 mon.b (mon.2) 23 : audit [DBG] from='mgr.? 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T07:17:44.805 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:44.015343+0000 mon.a (mon.0) 636 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-09-17T07:17:44.805 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:44.016085+0000 mon.b (mon.2) 24 : audit [DBG] from='mgr.? 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T07:17:44.805 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:44.016794+0000 mon.b (mon.2) 25 : audit [DBG] from='mgr.? 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T07:17:44.805 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:44.017053+0000 mon.b (mon.2) 26 : audit [DBG] from='mgr.? 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T07:17:44.806 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:43.543827+0000 mon.a (mon.0) 628 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:17:44.806 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:43.765465+0000 mon.a (mon.0) 629 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:17:44.806 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:43.769997+0000 mon.a (mon.0) 630 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:17:44.806 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:43.773409+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-17T07:17:44.806 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:43.778278+0000 mon.a (mon.0) 632 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-09-17T07:17:44.806 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:43.881348+0000 mon.a (mon.0) 633 : cluster [DBG] Standby manager daemon y started 2024-09-17T07:17:44.806 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:44.009432+0000 mon.a (mon.0) 634 : cluster [INF] Active manager daemon x restarted 2024-09-17T07:17:44.806 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:44.010279+0000 mon.a (mon.0) 635 : cluster [INF] Activating manager daemon x 2024-09-17T07:17:44.806 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:44.010391+0000 mon.b (mon.2) 23 : audit [DBG] from='mgr.? 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T07:17:44.807 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:44.015343+0000 mon.a (mon.0) 636 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-09-17T07:17:44.807 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:44.016085+0000 mon.b (mon.2) 24 : audit [DBG] from='mgr.? 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T07:17:44.807 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:44.016794+0000 mon.b (mon.2) 25 : audit [DBG] from='mgr.? 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T07:17:44.807 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:44.017053+0000 mon.b (mon.2) 26 : audit [DBG] from='mgr.? 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T07:17:44.807 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:43.543827+0000 mon.a (mon.0) 628 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:17:44.807 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:43.765465+0000 mon.a (mon.0) 629 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' 2024-09-17T07:17:44.807 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:43.769997+0000 mon.a (mon.0) 630 : audit [DBG] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:17:44.807 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:43.773409+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-17T07:17:44.808 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:43.778278+0000 mon.a (mon.0) 632 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-09-17T07:17:44.808 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:43.881348+0000 mon.a (mon.0) 633 : cluster [DBG] Standby manager daemon y started 2024-09-17T07:17:44.808 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:44.009432+0000 mon.a (mon.0) 634 : cluster [INF] Active manager daemon x restarted 2024-09-17T07:17:44.808 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:44.010279+0000 mon.a (mon.0) 635 : cluster [INF] Activating manager daemon x 2024-09-17T07:17:44.808 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:44.010391+0000 mon.b (mon.2) 23 : audit [DBG] from='mgr.? 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T07:17:44.808 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:44.015343+0000 mon.a (mon.0) 636 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-09-17T07:17:44.808 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:44.016085+0000 mon.b (mon.2) 24 : audit [DBG] from='mgr.? 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T07:17:44.808 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:44.016794+0000 mon.b (mon.2) 25 : audit [DBG] from='mgr.? 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T07:17:44.808 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:44.017053+0000 mon.b (mon.2) 26 : audit [DBG] from='mgr.? 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T07:17:45.074 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:17:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.y[23844]: ignoring --setuser ceph since I am not root 2024-09-17T07:17:45.075 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:17:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.y[23844]: ignoring --setgroup ceph since I am not root 2024-09-17T07:17:45.075 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:17:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.y[23844]: debug 2024-09-17T07:17:44.818+0000 7f400b89a700 1 -- 172.21.15.44:0/1984377483 <== mon.0 v2:172.21.15.44:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56433d6524e0 con 0x56433d629400 2024-09-17T07:17:45.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:17:43.770994+0000 mgr.y (mgr.14144) 380 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2024-09-17T07:17:45.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:17:43.773082+0000 mgr.y (mgr.14144) 381 : cephadm [INF] Failing over to other MGR 2024-09-17T07:17:45.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:44.478172+0000 mgr.y (mgr.14144) 382 : cluster [DBG] pgmap v332: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:45.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:44.775647+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-17T07:17:45.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:44.775735+0000 mon.a (mon.0) 638 : cluster [DBG] mgrmap e16: x(active, starting, since 0.765732s), standbys: y 2024-09-17T07:17:45.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:44.776506+0000 mon.b (mon.2) 27 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:17:45.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:44.776723+0000 mon.b (mon.2) 28 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:17:45.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:44.776888+0000 mon.b (mon.2) 29 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:17:45.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:44.777775+0000 mon.b (mon.2) 30 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T07:17:45.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:44.777953+0000 mon.b (mon.2) 31 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T07:17:45.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:44.778108+0000 mon.b (mon.2) 32 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T07:17:45.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:44.778313+0000 mon.b (mon.2) 33 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T07:17:45.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:44.778487+0000 mon.b (mon.2) 34 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:17:45.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:44.778670+0000 mon.b (mon.2) 35 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:17:45.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:44.778849+0000 mon.b (mon.2) 36 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:17:45.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:44.779041+0000 mon.b (mon.2) 37 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:17:45.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:17:43.770994+0000 mgr.y (mgr.14144) 380 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2024-09-17T07:17:45.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:17:43.773082+0000 mgr.y (mgr.14144) 381 : cephadm [INF] Failing over to other MGR 2024-09-17T07:17:45.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:44.478172+0000 mgr.y (mgr.14144) 382 : cluster [DBG] pgmap v332: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:45.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:44.775647+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-17T07:17:45.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:44.775735+0000 mon.a (mon.0) 638 : cluster [DBG] mgrmap e16: x(active, starting, since 0.765732s), standbys: y 2024-09-17T07:17:45.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:44.776506+0000 mon.b (mon.2) 27 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:17:45.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:44.776723+0000 mon.b (mon.2) 28 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:17:45.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:44.776888+0000 mon.b (mon.2) 29 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:17:45.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:44.777775+0000 mon.b (mon.2) 30 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T07:17:45.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:44.777953+0000 mon.b (mon.2) 31 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T07:17:45.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:44.778108+0000 mon.b (mon.2) 32 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T07:17:45.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:44.778313+0000 mon.b (mon.2) 33 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T07:17:45.828 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:44.778487+0000 mon.b (mon.2) 34 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:17:45.828 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:44.778670+0000 mon.b (mon.2) 35 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:17:45.828 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:44.778849+0000 mon.b (mon.2) 36 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:17:45.828 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:44.779041+0000 mon.b (mon.2) 37 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:17:45.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:44.779217+0000 mon.b (mon.2) 38 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:17:45.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:44.779409+0000 mon.b (mon.2) 39 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:17:45.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:44.780284+0000 mon.b (mon.2) 40 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T07:17:45.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:44.780485+0000 mon.b (mon.2) 41 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T07:17:45.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:44.781056+0000 mon.b (mon.2) 42 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T07:17:45.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:45.205828+0000 mon.a (mon.0) 639 : cluster [INF] Manager daemon x is now available 2024-09-17T07:17:45.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:45.222073+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:45.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:45.235404+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:45.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:45.242868+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:45.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:45.246990+0000 mon.b (mon.2) 43 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:17:45.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:45.271971+0000 mon.b (mon.2) 44 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:17:45.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:45.272377+0000 mon.a (mon.0) 643 : audit [INF] from='mgr.34814 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:17:45.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:45.283342+0000 mon.a (mon.0) 644 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:45.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:45.284319+0000 mon.b (mon.2) 45 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-17T07:17:45.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:45.284612+0000 mon.a (mon.0) 645 : audit [INF] from='mgr.34814 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-17T07:17:45.830 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:44.779217+0000 mon.b (mon.2) 38 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:17:45.830 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:44.779409+0000 mon.b (mon.2) 39 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:17:45.830 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:44.780284+0000 mon.b (mon.2) 40 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T07:17:45.830 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:44.780485+0000 mon.b (mon.2) 41 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T07:17:45.830 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:44.781056+0000 mon.b (mon.2) 42 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T07:17:45.830 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:45.205828+0000 mon.a (mon.0) 639 : cluster [INF] Manager daemon x is now available 2024-09-17T07:17:45.830 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:45.222073+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:45.830 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:45.235404+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:45.830 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:45.242868+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:45.830 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:45.246990+0000 mon.b (mon.2) 43 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:17:45.830 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:45.271971+0000 mon.b (mon.2) 44 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:17:45.830 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:45.272377+0000 mon.a (mon.0) 643 : audit [INF] from='mgr.34814 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:17:45.830 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:45.283342+0000 mon.a (mon.0) 644 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:45.831 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:45.284319+0000 mon.b (mon.2) 45 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-17T07:17:45.831 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:45.284612+0000 mon.a (mon.0) 645 : audit [INF] from='mgr.34814 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-17T07:17:45.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:17:43.770994+0000 mgr.y (mgr.14144) 380 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2024-09-17T07:17:45.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:17:43.773082+0000 mgr.y (mgr.14144) 381 : cephadm [INF] Failing over to other MGR 2024-09-17T07:17:45.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:44.478172+0000 mgr.y (mgr.14144) 382 : cluster [DBG] pgmap v332: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:45.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:44.775647+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14144 172.21.15.44:0/114684619' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-17T07:17:45.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:44.775735+0000 mon.a (mon.0) 638 : cluster [DBG] mgrmap e16: x(active, starting, since 0.765732s), standbys: y 2024-09-17T07:17:45.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:44.776506+0000 mon.b (mon.2) 27 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:17:45.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:44.776723+0000 mon.b (mon.2) 28 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:17:45.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:44.776888+0000 mon.b (mon.2) 29 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:17:45.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:44.777775+0000 mon.b (mon.2) 30 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T07:17:45.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:44.777953+0000 mon.b (mon.2) 31 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T07:17:45.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:44.778108+0000 mon.b (mon.2) 32 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T07:17:45.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:44.778313+0000 mon.b (mon.2) 33 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T07:17:45.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:44.778487+0000 mon.b (mon.2) 34 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:17:45.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:44.778670+0000 mon.b (mon.2) 35 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:17:45.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:44.778849+0000 mon.b (mon.2) 36 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:17:45.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:44.779041+0000 mon.b (mon.2) 37 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:17:45.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:44.779217+0000 mon.b (mon.2) 38 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:17:45.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:44.779409+0000 mon.b (mon.2) 39 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:17:45.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:44.780284+0000 mon.b (mon.2) 40 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T07:17:45.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:44.780485+0000 mon.b (mon.2) 41 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T07:17:45.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:44.781056+0000 mon.b (mon.2) 42 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T07:17:45.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:45.205828+0000 mon.a (mon.0) 639 : cluster [INF] Manager daemon x is now available 2024-09-17T07:17:45.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:45.222073+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:45.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:45.235404+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:45.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:45.242868+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:45.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:45.246990+0000 mon.b (mon.2) 43 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:17:45.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:45.271971+0000 mon.b (mon.2) 44 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:17:45.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:45.272377+0000 mon.a (mon.0) 643 : audit [INF] from='mgr.34814 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:17:45.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:45.283342+0000 mon.a (mon.0) 644 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:45.867 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:45.284319+0000 mon.b (mon.2) 45 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-17T07:17:45.867 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:45.284612+0000 mon.a (mon.0) 645 : audit [INF] from='mgr.34814 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-17T07:17:47.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:17:45.215784+0000 mgr.x (mgr.34814) 1 : cephadm [INF] inventory: adjusted host smithi044 addr 'smithi044' -> '172.21.15.44' 2024-09-17T07:17:47.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:17:45.235837+0000 mgr.x (mgr.34814) 2 : cephadm [INF] Queued rgw.foo for migration 2024-09-17T07:17:47.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:17:45.243258+0000 mgr.x (mgr.34814) 3 : cephadm [INF] Queued rgw.smpl for migration 2024-09-17T07:17:47.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:17:45.711995+0000 mgr.x (mgr.34814) 4 : cephadm [INF] Deploying cephadm binary to smithi179 2024-09-17T07:17:47.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:17:45.730817+0000 mgr.x (mgr.34814) 5 : cephadm [INF] Deploying cephadm binary to smithi044 2024-09-17T07:17:47.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:45.841760+0000 mon.a (mon.0) 646 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:47.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:45.856230+0000 mon.a (mon.0) 647 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:47.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:46.027315+0000 mon.a (mon.0) 648 : cluster [DBG] mgrmap e17: x(active, since 2s), standbys: y 2024-09-17T07:17:47.115 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:17:46.855+0000 7f1f56323640 -1 log_channel(cephadm) log [ERR] : [17/Sep/2024:07:17:46] ENGINE Error in HTTPServer.serve 2024-09-17T07:17:47.115 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: Traceback (most recent call last): 2024-09-17T07:17:47.115 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T07:17:47.115 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: self._connections.run(self.expiration_interval) 2024-09-17T07:17:47.115 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T07:17:47.115 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: self._run(expiration_interval) 2024-09-17T07:17:47.115 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T07:17:47.116 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T07:17:47.116 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T07:17:47.116 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T07:17:47.116 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T07:17:47.116 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: s = self.context.wrap_socket( 2024-09-17T07:17:47.116 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T07:17:47.116 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: return self.sslsocket_class._create( 2024-09-17T07:17:47.116 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T07:17:47.116 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: self.do_handshake() 2024-09-17T07:17:47.116 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T07:17:47.116 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: self._sslobj.do_handshake() 2024-09-17T07:17:47.116 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T07:17:47.116 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:17:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: 2024-09-17T07:17:47.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:17:45.215784+0000 mgr.x (mgr.34814) 1 : cephadm [INF] inventory: adjusted host smithi044 addr 'smithi044' -> '172.21.15.44' 2024-09-17T07:17:47.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:17:45.235837+0000 mgr.x (mgr.34814) 2 : cephadm [INF] Queued rgw.foo for migration 2024-09-17T07:17:47.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:17:45.243258+0000 mgr.x (mgr.34814) 3 : cephadm [INF] Queued rgw.smpl for migration 2024-09-17T07:17:47.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:17:45.711995+0000 mgr.x (mgr.34814) 4 : cephadm [INF] Deploying cephadm binary to smithi179 2024-09-17T07:17:47.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:17:45.730817+0000 mgr.x (mgr.34814) 5 : cephadm [INF] Deploying cephadm binary to smithi044 2024-09-17T07:17:47.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:45.841760+0000 mon.a (mon.0) 646 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:47.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:45.856230+0000 mon.a (mon.0) 647 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:47.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:46.027315+0000 mon.a (mon.0) 648 : cluster [DBG] mgrmap e17: x(active, since 2s), standbys: y 2024-09-17T07:17:47.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:17:45.215784+0000 mgr.x (mgr.34814) 1 : cephadm [INF] inventory: adjusted host smithi044 addr 'smithi044' -> '172.21.15.44' 2024-09-17T07:17:47.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:17:45.235837+0000 mgr.x (mgr.34814) 2 : cephadm [INF] Queued rgw.foo for migration 2024-09-17T07:17:47.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:17:45.243258+0000 mgr.x (mgr.34814) 3 : cephadm [INF] Queued rgw.smpl for migration 2024-09-17T07:17:47.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:17:45.711995+0000 mgr.x (mgr.34814) 4 : cephadm [INF] Deploying cephadm binary to smithi179 2024-09-17T07:17:47.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:17:45.730817+0000 mgr.x (mgr.34814) 5 : cephadm [INF] Deploying cephadm binary to smithi044 2024-09-17T07:17:47.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:45.841760+0000 mon.a (mon.0) 646 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:47.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:45.856230+0000 mon.a (mon.0) 647 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:47.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:46.027315+0000 mon.a (mon.0) 648 : cluster [DBG] mgrmap e17: x(active, since 2s), standbys: y 2024-09-17T07:17:47.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:46.045376+0000 mgr.x (mgr.34814) 6 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:47.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:17:46.737672+0000 mgr.x (mgr.34814) 7 : cephadm [INF] [17/Sep/2024:07:17:46] ENGINE Bus STARTING 2024-09-17T07:17:47.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:46.780435+0000 mgr.x (mgr.34814) 8 : cluster [DBG] pgmap v4: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:47.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:17:46.852876+0000 mgr.x (mgr.34814) 9 : cephadm [INF] [17/Sep/2024:07:17:46] ENGINE Serving on https://172.21.15.179:7150 2024-09-17T07:17:47.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:17:46.856121+0000 mgr.x (mgr.34814) 10 : cephadm [ERR] [17/Sep/2024:07:17:46] ENGINE Error in HTTPServer.serve 2024-09-17T07:17:47.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: Traceback (most recent call last): 2024-09-17T07:17:47.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T07:17:47.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: self._connections.run(self.expiration_interval) 2024-09-17T07:17:47.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T07:17:47.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: self._run(expiration_interval) 2024-09-17T07:17:47.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T07:17:47.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T07:17:47.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T07:17:47.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T07:17:47.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T07:17:47.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: s = self.context.wrap_socket( 2024-09-17T07:17:47.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T07:17:47.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: return self.sslsocket_class._create( 2024-09-17T07:17:47.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T07:17:47.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: self.do_handshake() 2024-09-17T07:17:47.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T07:17:47.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: self._sslobj.do_handshake() 2024-09-17T07:17:47.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T07:17:47.867 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: 2024-09-17T07:17:47.867 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:17:46.954287+0000 mgr.x (mgr.34814) 11 : cephadm [INF] [17/Sep/2024:07:17:46] ENGINE Serving on http://172.21.15.179:8765 2024-09-17T07:17:47.867 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:17:46.954505+0000 mgr.x (mgr.34814) 12 : cephadm [INF] [17/Sep/2024:07:17:46] ENGINE Bus STARTED 2024-09-17T07:17:47.867 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:47.021926+0000 mon.a (mon.0) 649 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-17T07:17:47.867 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:47.021966+0000 mon.a (mon.0) 650 : cluster [INF] Cluster is now healthy 2024-09-17T07:17:47.867 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:47.256771+0000 mon.b (mon.2) 46 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:17:47.867 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:47.257268+0000 mon.a (mon.0) 651 : audit [INF] from='mgr.34814 ' entity='mgr.x' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:17:48.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:46.045376+0000 mgr.x (mgr.34814) 6 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:48.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:17:46.737672+0000 mgr.x (mgr.34814) 7 : cephadm [INF] [17/Sep/2024:07:17:46] ENGINE Bus STARTING 2024-09-17T07:17:48.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:46.780435+0000 mgr.x (mgr.34814) 8 : cluster [DBG] pgmap v4: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:48.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:17:46.852876+0000 mgr.x (mgr.34814) 9 : cephadm [INF] [17/Sep/2024:07:17:46] ENGINE Serving on https://172.21.15.179:7150 2024-09-17T07:17:48.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:17:46.856121+0000 mgr.x (mgr.34814) 10 : cephadm [ERR] [17/Sep/2024:07:17:46] ENGINE Error in HTTPServer.serve 2024-09-17T07:17:48.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: Traceback (most recent call last): 2024-09-17T07:17:48.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T07:17:48.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: self._connections.run(self.expiration_interval) 2024-09-17T07:17:48.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T07:17:48.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: self._run(expiration_interval) 2024-09-17T07:17:48.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T07:17:48.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T07:17:48.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T07:17:48.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T07:17:48.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T07:17:48.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: s = self.context.wrap_socket( 2024-09-17T07:17:48.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T07:17:48.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: return self.sslsocket_class._create( 2024-09-17T07:17:48.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T07:17:48.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: self.do_handshake() 2024-09-17T07:17:48.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T07:17:48.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: self._sslobj.do_handshake() 2024-09-17T07:17:48.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T07:17:48.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: 2024-09-17T07:17:48.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:17:46.954287+0000 mgr.x (mgr.34814) 11 : cephadm [INF] [17/Sep/2024:07:17:46] ENGINE Serving on http://172.21.15.179:8765 2024-09-17T07:17:48.327 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:46.045376+0000 mgr.x (mgr.34814) 6 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:48.328 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:17:46.737672+0000 mgr.x (mgr.34814) 7 : cephadm [INF] [17/Sep/2024:07:17:46] ENGINE Bus STARTING 2024-09-17T07:17:48.328 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:46.780435+0000 mgr.x (mgr.34814) 8 : cluster [DBG] pgmap v4: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:48.328 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:17:46.852876+0000 mgr.x (mgr.34814) 9 : cephadm [INF] [17/Sep/2024:07:17:46] ENGINE Serving on https://172.21.15.179:7150 2024-09-17T07:17:48.328 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:17:46.856121+0000 mgr.x (mgr.34814) 10 : cephadm [ERR] [17/Sep/2024:07:17:46] ENGINE Error in HTTPServer.serve 2024-09-17T07:17:48.328 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: Traceback (most recent call last): 2024-09-17T07:17:48.328 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T07:17:48.328 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: self._connections.run(self.expiration_interval) 2024-09-17T07:17:48.328 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T07:17:48.328 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: self._run(expiration_interval) 2024-09-17T07:17:48.328 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T07:17:48.328 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T07:17:48.328 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T07:17:48.329 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T07:17:48.329 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T07:17:48.329 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: s = self.context.wrap_socket( 2024-09-17T07:17:48.329 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T07:17:48.329 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: return self.sslsocket_class._create( 2024-09-17T07:17:48.329 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T07:17:48.329 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: self.do_handshake() 2024-09-17T07:17:48.329 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T07:17:48.329 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: self._sslobj.do_handshake() 2024-09-17T07:17:48.329 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T07:17:48.329 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: 2024-09-17T07:17:48.329 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:17:46.954287+0000 mgr.x (mgr.34814) 11 : cephadm [INF] [17/Sep/2024:07:17:46] ENGINE Serving on http://172.21.15.179:8765 2024-09-17T07:17:48.330 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:17:46.954505+0000 mgr.x (mgr.34814) 12 : cephadm [INF] [17/Sep/2024:07:17:46] ENGINE Bus STARTED 2024-09-17T07:17:48.330 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:47.021926+0000 mon.a (mon.0) 649 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-17T07:17:48.330 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:47.021966+0000 mon.a (mon.0) 650 : cluster [INF] Cluster is now healthy 2024-09-17T07:17:48.330 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:47.256771+0000 mon.b (mon.2) 46 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:17:48.330 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:47.257268+0000 mon.a (mon.0) 651 : audit [INF] from='mgr.34814 ' entity='mgr.x' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:17:48.330 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:17:46.954505+0000 mgr.x (mgr.34814) 12 : cephadm [INF] [17/Sep/2024:07:17:46] ENGINE Bus STARTED 2024-09-17T07:17:48.330 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:47.021926+0000 mon.a (mon.0) 649 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-17T07:17:48.330 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:47.021966+0000 mon.a (mon.0) 650 : cluster [INF] Cluster is now healthy 2024-09-17T07:17:48.330 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:47.256771+0000 mon.b (mon.2) 46 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:17:48.331 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:47.257268+0000 mon.a (mon.0) 651 : audit [INF] from='mgr.34814 ' entity='mgr.x' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:17:49.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:48.031869+0000 mon.a (mon.0) 652 : audit [INF] from='mgr.34814 ' entity='mgr.x' cmd='[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]': finished 2024-09-17T07:17:49.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:48.032087+0000 mon.a (mon.0) 653 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-09-17T07:17:49.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:48.033103+0000 mon.b (mon.2) 47 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:17:49.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:48.033489+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.34814 ' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:17:49.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:48.780773+0000 mgr.x (mgr.34814) 13 : cluster [DBG] pgmap v6: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:49.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:48.031869+0000 mon.a (mon.0) 652 : audit [INF] from='mgr.34814 ' entity='mgr.x' cmd='[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]': finished 2024-09-17T07:17:49.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:48.032087+0000 mon.a (mon.0) 653 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-09-17T07:17:49.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:48.033103+0000 mon.b (mon.2) 47 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:17:49.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:48.033489+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.34814 ' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:17:49.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:48.780773+0000 mgr.x (mgr.34814) 13 : cluster [DBG] pgmap v6: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:49.345 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:48.031869+0000 mon.a (mon.0) 652 : audit [INF] from='mgr.34814 ' entity='mgr.x' cmd='[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]': finished 2024-09-17T07:17:49.345 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:48.032087+0000 mon.a (mon.0) 653 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-09-17T07:17:49.345 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:48.033103+0000 mon.b (mon.2) 47 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:17:49.345 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:48.033489+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.34814 ' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T07:17:49.345 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:48.780773+0000 mgr.x (mgr.34814) 13 : cluster [DBG] pgmap v6: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:50.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:49.035965+0000 mon.a (mon.0) 655 : audit [INF] from='mgr.34814 ' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-17T07:17:50.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:49.036046+0000 mon.a (mon.0) 656 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-09-17T07:17:50.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:49.035965+0000 mon.a (mon.0) 655 : audit [INF] from='mgr.34814 ' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-17T07:17:50.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:49.036046+0000 mon.a (mon.0) 656 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-09-17T07:17:50.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:49.035965+0000 mon.a (mon.0) 655 : audit [INF] from='mgr.34814 ' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-17T07:17:50.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:49.036046+0000 mon.a (mon.0) 656 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-09-17T07:17:51.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:50.044987+0000 mon.a (mon.0) 657 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:51.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:50.255585+0000 mon.a (mon.0) 658 : cluster [DBG] Standby manager daemon y restarted 2024-09-17T07:17:51.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:50.255758+0000 mon.a (mon.0) 659 : cluster [DBG] Standby manager daemon y started 2024-09-17T07:17:51.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:50.256889+0000 mon.a (mon.0) 660 : audit [DBG] from='mgr.? 172.21.15.44:0/3121965223' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-17T07:17:51.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:50.258689+0000 mon.a (mon.0) 661 : audit [DBG] from='mgr.? 172.21.15.44:0/3121965223' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T07:17:51.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:50.259783+0000 mon.a (mon.0) 662 : audit [DBG] from='mgr.? 172.21.15.44:0/3121965223' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-17T07:17:51.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:50.260164+0000 mon.a (mon.0) 663 : audit [DBG] from='mgr.? 172.21.15.44:0/3121965223' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T07:17:51.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:50.701363+0000 mon.a (mon.0) 664 : cluster [DBG] mgrmap e18: x(active, since 6s), standbys: y 2024-09-17T07:17:51.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:50.713493+0000 mon.a (mon.0) 665 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:51.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:50.718791+0000 mon.a (mon.0) 666 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:51.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:50.781101+0000 mgr.x (mgr.34814) 14 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:51.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:50.044987+0000 mon.a (mon.0) 657 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:51.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:50.255585+0000 mon.a (mon.0) 658 : cluster [DBG] Standby manager daemon y restarted 2024-09-17T07:17:51.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:50.255758+0000 mon.a (mon.0) 659 : cluster [DBG] Standby manager daemon y started 2024-09-17T07:17:51.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:50.256889+0000 mon.a (mon.0) 660 : audit [DBG] from='mgr.? 172.21.15.44:0/3121965223' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-17T07:17:51.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:50.258689+0000 mon.a (mon.0) 661 : audit [DBG] from='mgr.? 172.21.15.44:0/3121965223' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T07:17:51.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:50.259783+0000 mon.a (mon.0) 662 : audit [DBG] from='mgr.? 172.21.15.44:0/3121965223' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-17T07:17:51.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:50.260164+0000 mon.a (mon.0) 663 : audit [DBG] from='mgr.? 172.21.15.44:0/3121965223' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T07:17:51.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:50.701363+0000 mon.a (mon.0) 664 : cluster [DBG] mgrmap e18: x(active, since 6s), standbys: y 2024-09-17T07:17:51.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:50.713493+0000 mon.a (mon.0) 665 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:51.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:50.718791+0000 mon.a (mon.0) 666 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:51.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:50.781101+0000 mgr.x (mgr.34814) 14 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:51.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:50.044987+0000 mon.a (mon.0) 657 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:51.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:50.255585+0000 mon.a (mon.0) 658 : cluster [DBG] Standby manager daemon y restarted 2024-09-17T07:17:51.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:50.255758+0000 mon.a (mon.0) 659 : cluster [DBG] Standby manager daemon y started 2024-09-17T07:17:51.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:50.256889+0000 mon.a (mon.0) 660 : audit [DBG] from='mgr.? 172.21.15.44:0/3121965223' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-17T07:17:51.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:50.258689+0000 mon.a (mon.0) 661 : audit [DBG] from='mgr.? 172.21.15.44:0/3121965223' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T07:17:51.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:50.259783+0000 mon.a (mon.0) 662 : audit [DBG] from='mgr.? 172.21.15.44:0/3121965223' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-17T07:17:51.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:50.260164+0000 mon.a (mon.0) 663 : audit [DBG] from='mgr.? 172.21.15.44:0/3121965223' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T07:17:51.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:50.701363+0000 mon.a (mon.0) 664 : cluster [DBG] mgrmap e18: x(active, since 6s), standbys: y 2024-09-17T07:17:51.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:50.713493+0000 mon.a (mon.0) 665 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:51.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:50.718791+0000 mon.a (mon.0) 666 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:51.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:50.781101+0000 mgr.x (mgr.34814) 14 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:17:52.529 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:51.237223+0000 mon.a (mon.0) 667 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:52.529 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:51.242666+0000 mon.a (mon.0) 668 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:52.529 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:51.671040+0000 mon.a (mon.0) 669 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:52.530 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:51.677110+0000 mon.a (mon.0) 670 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:52.530 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:52.172293+0000 mon.a (mon.0) 671 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:52.530 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:52.179012+0000 mon.a (mon.0) 672 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:52.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:51.237223+0000 mon.a (mon.0) 667 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:52.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:51.242666+0000 mon.a (mon.0) 668 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:52.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:51.671040+0000 mon.a (mon.0) 669 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:52.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:51.677110+0000 mon.a (mon.0) 670 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:52.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:52.172293+0000 mon.a (mon.0) 671 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:52.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:52.179012+0000 mon.a (mon.0) 672 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:52.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:51.237223+0000 mon.a (mon.0) 667 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:52.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:51.242666+0000 mon.a (mon.0) 668 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:52.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:51.671040+0000 mon.a (mon.0) 669 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:52.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:51.677110+0000 mon.a (mon.0) 670 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:52.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:52.172293+0000 mon.a (mon.0) 671 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:52.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:52.179012+0000 mon.a (mon.0) 672 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:53.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:53 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:52.782007+0000 mgr.x (mgr.34814) 15 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 242 KiB/s wr, 32 op/s 2024-09-17T07:17:53.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:53 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:52.782007+0000 mgr.x (mgr.34814) 15 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 242 KiB/s wr, 32 op/s 2024-09-17T07:17:53.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:52.782007+0000 mgr.x (mgr.34814) 15 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 242 KiB/s wr, 32 op/s 2024-09-17T07:17:54.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:53.313874+0000 mon.a (mon.0) 673 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:54.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:53.313874+0000 mon.a (mon.0) 673 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:54.613 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:53.313874+0000 mon.a (mon.0) 673 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:55.982 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:54.655960+0000 mon.a (mon.0) 674 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:55.982 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:54.663502+0000 mon.a (mon.0) 675 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:55.982 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:54.823178+0000 mon.a (mon.0) 676 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T07:17:56.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:54.655960+0000 mon.a (mon.0) 674 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:56.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:54.663502+0000 mon.a (mon.0) 675 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:56.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:54.823178+0000 mon.a (mon.0) 676 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T07:17:56.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:54.655960+0000 mon.a (mon.0) 674 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:56.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:54.663502+0000 mon.a (mon.0) 675 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:17:56.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:54.823178+0000 mon.a (mon.0) 676 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T07:17:57.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:54.782356+0000 mgr.x (mgr.34814) 16 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 204 KiB/s wr, 27 op/s 2024-09-17T07:17:57.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:54.823416+0000 mon.b (mon.2) 48 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:17:57.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:54.823607+0000 mon.b (mon.2) 49 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:17:57.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:54.823753+0000 mon.b (mon.2) 50 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:17:57.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:55.718187+0000 mon.a (mon.0) 677 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T07:17:57.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:55.721952+0000 mon.c (mon.1) 32 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T07:17:57.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:55.722860+0000 mon.b (mon.2) 51 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:17:57.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:55.723075+0000 mon.b (mon.2) 52 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:17:57.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:55.723442+0000 mon.b (mon.2) 53 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:17:57.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:56.610979+0000 mon.c (mon.1) 33 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T07:17:57.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:54.782356+0000 mgr.x (mgr.34814) 16 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 204 KiB/s wr, 27 op/s 2024-09-17T07:17:57.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:54.823416+0000 mon.b (mon.2) 48 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:17:57.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:54.823607+0000 mon.b (mon.2) 49 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:17:57.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:54.823753+0000 mon.b (mon.2) 50 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:17:57.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:55.718187+0000 mon.a (mon.0) 677 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T07:17:57.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:55.721952+0000 mon.c (mon.1) 32 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T07:17:57.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:55.722860+0000 mon.b (mon.2) 51 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:17:57.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:55.723075+0000 mon.b (mon.2) 52 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:17:57.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:55.723442+0000 mon.b (mon.2) 53 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:17:57.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:56.610979+0000 mon.c (mon.1) 33 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T07:17:57.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:54.782356+0000 mgr.x (mgr.34814) 16 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 204 KiB/s wr, 27 op/s 2024-09-17T07:17:57.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:54.823416+0000 mon.b (mon.2) 48 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:17:57.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:54.823607+0000 mon.b (mon.2) 49 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:17:57.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:54.823753+0000 mon.b (mon.2) 50 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:17:57.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:55.718187+0000 mon.a (mon.0) 677 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T07:17:57.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:55.721952+0000 mon.c (mon.1) 32 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T07:17:57.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:55.722860+0000 mon.b (mon.2) 51 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:17:57.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:55.723075+0000 mon.b (mon.2) 52 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:17:57.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:55.723442+0000 mon.b (mon.2) 53 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:17:57.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:56.610979+0000 mon.c (mon.1) 33 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T07:17:58.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:56.616491+0000 mon.b (mon.2) 54 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T07:17:58.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:56.782982+0000 mgr.x (mgr.34814) 17 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 246 KiB/s wr, 28 op/s 2024-09-17T07:17:58.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:57.113655+0000 mon.b (mon.2) 55 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T07:17:58.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:57.114532+0000 mon.b (mon.2) 56 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:17:58.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:57.114671+0000 mon.b (mon.2) 57 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:17:58.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:17:57.114808+0000 mon.b (mon.2) 58 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:17:58.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:56.616491+0000 mon.b (mon.2) 54 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T07:17:58.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:56.782982+0000 mgr.x (mgr.34814) 17 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 246 KiB/s wr, 28 op/s 2024-09-17T07:17:58.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:57.113655+0000 mon.b (mon.2) 55 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T07:17:58.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:57.114532+0000 mon.b (mon.2) 56 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:17:58.577 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:57.114671+0000 mon.b (mon.2) 57 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:17:58.577 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:17:57.114808+0000 mon.b (mon.2) 58 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:17:58.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:58 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:56.616491+0000 mon.b (mon.2) 54 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T07:17:58.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:58 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:56.782982+0000 mgr.x (mgr.34814) 17 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 246 KiB/s wr, 28 op/s 2024-09-17T07:17:58.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:58 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:57.113655+0000 mon.b (mon.2) 55 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T07:17:58.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:58 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:57.114532+0000 mon.b (mon.2) 56 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:17:58.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:58 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:57.114671+0000 mon.b (mon.2) 57 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:17:58.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:58 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:17:57.114808+0000 mon.b (mon.2) 58 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:17:59.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:17:59 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:17:58.783552+0000 mgr.x (mgr.34814) 18 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 215 KiB/s wr, 25 op/s 2024-09-17T07:17:59.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:17:59 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:17:58.783552+0000 mgr.x (mgr.34814) 18 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 215 KiB/s wr, 25 op/s 2024-09-17T07:17:59.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:17:59 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:17:58.783552+0000 mgr.x (mgr.34814) 18 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 215 KiB/s wr, 25 op/s 2024-09-17T07:18:02.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:01 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:18:00.784117+0000 mgr.x (mgr.34814) 19 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 7.7 KiB/s rd, 183 KiB/s wr, 21 op/s 2024-09-17T07:18:02.324 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:18:00.784117+0000 mgr.x (mgr.34814) 19 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 7.7 KiB/s rd, 183 KiB/s wr, 21 op/s 2024-09-17T07:18:02.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:18:00.784117+0000 mgr.x (mgr.34814) 19 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 7.7 KiB/s rd, 183 KiB/s wr, 21 op/s 2024-09-17T07:18:04.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:03 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:18:02.784791+0000 mgr.x (mgr.34814) 20 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 8.3 KiB/s rd, 201 KiB/s wr, 22 op/s 2024-09-17T07:18:04.324 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:03 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:18:02.784791+0000 mgr.x (mgr.34814) 20 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 8.3 KiB/s rd, 201 KiB/s wr, 22 op/s 2024-09-17T07:18:04.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:03 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:18:02.784791+0000 mgr.x (mgr.34814) 20 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 8.3 KiB/s rd, 201 KiB/s wr, 22 op/s 2024-09-17T07:18:05.825 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:05 smithi044 systemd[1]: Stopping Ceph node-exporter.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:18:05.825 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:05 smithi044 podman[51282]: 2024-09-17 07:18:05.58737157 +0000 UTC m=+0.120791214 container died d7284955755c8b5cf23641b1955969aabd7589f2e270d493c7628f8033ad258f (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a, maintainer=The Prometheus Authors ) 2024-09-17T07:18:05.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:18:04.785222+0000 mgr.x (mgr.34814) 21 : cluster [DBG] pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-17T07:18:05.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:05.055496+0000 mon.a (mon.0) 678 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:05.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:05.062796+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:05.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:05.069347+0000 mon.a (mon.0) 680 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:05.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:05.074503+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:05.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:05.079839+0000 mon.a (mon.0) 682 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:05.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:05.086013+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:05.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:05.094911+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:05.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:05.102571+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:05.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:05.188300+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:05.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:05.191335+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:05.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:05.194442+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:05.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:05.197826+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:05.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:18:04.785222+0000 mgr.x (mgr.34814) 21 : cluster [DBG] pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-17T07:18:05.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:05.055496+0000 mon.a (mon.0) 678 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:05.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:05.062796+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:05.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:05.069347+0000 mon.a (mon.0) 680 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:05.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:05.074503+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:05.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:05.079839+0000 mon.a (mon.0) 682 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:05.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:05.086013+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:05.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:05.094911+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:05.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:05.102571+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:05.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:05.188300+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:05.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:05.191335+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:05.828 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:05.194442+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:05.828 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:05.197826+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:06.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:05 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:18:04.785222+0000 mgr.x (mgr.34814) 21 : cluster [DBG] pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-17T07:18:06.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:05 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:05.055496+0000 mon.a (mon.0) 678 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:06.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:05 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:05.062796+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:06.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:05 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:05.069347+0000 mon.a (mon.0) 680 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:06.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:05 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:05.074503+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:06.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:05 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:05.079839+0000 mon.a (mon.0) 682 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:06.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:05 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:05.086013+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:06.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:05 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:05.094911+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:06.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:05 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:05.102571+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:06.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:05 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:05.188300+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:06.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:05 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:05.191335+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:06.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:05 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:05.194442+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:06.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:05 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:05.197826+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:06.324 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 podman[51282]: 2024-09-17 07:18:06.035979086 +0000 UTC m=+0.569398733 container cleanup d7284955755c8b5cf23641b1955969aabd7589f2e270d493c7628f8033ad258f (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a, maintainer=The Prometheus Authors ) 2024-09-17T07:18:06.325 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 podman[51282]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a 2024-09-17T07:18:06.665 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 podman[51294]: 2024-09-17 07:18:06.345884689 +0000 UTC m=+0.757926165 container remove d7284955755c8b5cf23641b1955969aabd7589f2e270d493c7628f8033ad258f (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a, maintainer=The Prometheus Authors ) 2024-09-17T07:18:06.665 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-17T07:18:06.665 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-17T07:18:06.665 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 systemd[1]: Stopped Ceph node-exporter.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:18:06.665 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@node-exporter.a.service: Consumed 1.307s CPU time. 2024-09-17T07:18:06.665 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 systemd[1]: Starting Ceph node-exporter.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:18:06.665 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 podman[51334]: 2024-09-17 07:18:06.437294097 +0000 UTC m=+0.016559925 image pull e5a616e4b9cf68dfcad7782b78e118be4310022e874d52da85c55923fb615f87 docker.io/prom/node-exporter:v0.18.1 2024-09-17T07:18:06.665 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 podman[51334]: 2024-09-17 07:18:06.538788435 +0000 UTC m=+0.118054295 container create 8a2cde5bd9ed5da1b1583a93a47a04eba639f455650688b7108ef5d73ca66027 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a, maintainer=The Prometheus Authors ) 2024-09-17T07:18:07.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:18:05.064987+0000 mgr.x (mgr.34814) 22 : cluster [DBG] pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 76 KiB/s wr, 5 op/s 2024-09-17T07:18:07.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:18:05.098112+0000 mgr.x (mgr.34814) 23 : cephadm [INF] No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-09-17T07:18:07.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:18:05.098286+0000 mgr.x (mgr.34814) 24 : cephadm [INF] No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-09-17T07:18:07.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:18:05.198532+0000 mgr.x (mgr.34814) 25 : cephadm [INF] Reconfiguring node-exporter.a (dependencies changed)... 2024-09-17T07:18:07.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:18:05.198837+0000 mgr.x (mgr.34814) 26 : cephadm [INF] Reconfiguring daemon node-exporter.a on smithi044 2024-09-17T07:18:07.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:18:05.690884+0000 mon.a (mon.0) 690 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T07:18:07.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:18:05.064987+0000 mgr.x (mgr.34814) 22 : cluster [DBG] pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 76 KiB/s wr, 5 op/s 2024-09-17T07:18:07.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:18:05.098112+0000 mgr.x (mgr.34814) 23 : cephadm [INF] No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-09-17T07:18:07.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:18:05.098286+0000 mgr.x (mgr.34814) 24 : cephadm [INF] No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-09-17T07:18:07.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:18:05.198532+0000 mgr.x (mgr.34814) 25 : cephadm [INF] Reconfiguring node-exporter.a (dependencies changed)... 2024-09-17T07:18:07.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:18:05.198837+0000 mgr.x (mgr.34814) 26 : cephadm [INF] Reconfiguring daemon node-exporter.a on smithi044 2024-09-17T07:18:07.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:18:05.690884+0000 mon.a (mon.0) 690 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T07:18:07.076 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 podman[51334]: 2024-09-17 07:18:06.77411652 +0000 UTC m=+0.353382338 container init 8a2cde5bd9ed5da1b1583a93a47a04eba639f455650688b7108ef5d73ca66027 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a, maintainer=The Prometheus Authors ) 2024-09-17T07:18:07.076 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 podman[51334]: 2024-09-17 07:18:06.777191547 +0000 UTC m=+0.356457366 container start 8a2cde5bd9ed5da1b1583a93a47a04eba639f455650688b7108ef5d73ca66027 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a, maintainer=The Prometheus Authors ) 2024-09-17T07:18:07.076 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[51349]: time="2024-09-17T07:18:06Z" level=info msg="Starting node_exporter (version=0.18.1, branch=HEAD, revision=3db77732e925c08f675d7404a8c46466b2ece83e)" source="node_exporter.go:156" 2024-09-17T07:18:07.076 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[51349]: time="2024-09-17T07:18:06Z" level=info msg="Build context (go=go1.12.5, user=root@b50852a1acba, date=20190604-16:41:18)" source="node_exporter.go:157" 2024-09-17T07:18:07.076 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[51349]: time="2024-09-17T07:18:06Z" level=info msg="Enabled collectors:" source="node_exporter.go:97" 2024-09-17T07:18:07.076 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[51349]: time="2024-09-17T07:18:06Z" level=info msg=" - arp" source="node_exporter.go:104" 2024-09-17T07:18:07.076 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[51349]: time="2024-09-17T07:18:06Z" level=info msg=" - bcache" source="node_exporter.go:104" 2024-09-17T07:18:07.076 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[51349]: time="2024-09-17T07:18:06Z" level=info msg=" - bonding" source="node_exporter.go:104" 2024-09-17T07:18:07.076 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[51349]: time="2024-09-17T07:18:06Z" level=info msg=" - conntrack" source="node_exporter.go:104" 2024-09-17T07:18:07.077 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[51349]: time="2024-09-17T07:18:06Z" level=info msg=" - cpu" source="node_exporter.go:104" 2024-09-17T07:18:07.077 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[51349]: time="2024-09-17T07:18:06Z" level=info msg=" - cpufreq" source="node_exporter.go:104" 2024-09-17T07:18:07.077 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[51349]: time="2024-09-17T07:18:06Z" level=info msg=" - diskstats" source="node_exporter.go:104" 2024-09-17T07:18:07.077 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[51349]: time="2024-09-17T07:18:06Z" level=info msg=" - edac" source="node_exporter.go:104" 2024-09-17T07:18:07.077 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[51349]: time="2024-09-17T07:18:06Z" level=info msg=" - entropy" source="node_exporter.go:104" 2024-09-17T07:18:07.077 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[51349]: time="2024-09-17T07:18:06Z" level=info msg=" - filefd" source="node_exporter.go:104" 2024-09-17T07:18:07.077 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[51349]: time="2024-09-17T07:18:06Z" level=info msg=" - filesystem" source="node_exporter.go:104" 2024-09-17T07:18:07.077 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[51349]: time="2024-09-17T07:18:06Z" level=info msg=" - hwmon" source="node_exporter.go:104" 2024-09-17T07:18:07.077 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[51349]: time="2024-09-17T07:18:06Z" level=info msg=" - infiniband" source="node_exporter.go:104" 2024-09-17T07:18:07.077 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[51349]: time="2024-09-17T07:18:06Z" level=info msg=" - ipvs" source="node_exporter.go:104" 2024-09-17T07:18:07.077 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[51349]: time="2024-09-17T07:18:06Z" level=info msg=" - loadavg" source="node_exporter.go:104" 2024-09-17T07:18:07.077 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[51349]: time="2024-09-17T07:18:06Z" level=info msg=" - mdadm" source="node_exporter.go:104" 2024-09-17T07:18:07.077 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[51349]: time="2024-09-17T07:18:06Z" level=info msg=" - meminfo" source="node_exporter.go:104" 2024-09-17T07:18:07.077 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[51349]: time="2024-09-17T07:18:06Z" level=info msg=" - netclass" source="node_exporter.go:104" 2024-09-17T07:18:07.078 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[51349]: time="2024-09-17T07:18:06Z" level=info msg=" - netdev" source="node_exporter.go:104" 2024-09-17T07:18:07.078 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[51349]: time="2024-09-17T07:18:06Z" level=info msg=" - netstat" source="node_exporter.go:104" 2024-09-17T07:18:07.078 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[51349]: time="2024-09-17T07:18:06Z" level=info msg=" - nfs" source="node_exporter.go:104" 2024-09-17T07:18:07.078 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[51349]: time="2024-09-17T07:18:06Z" level=info msg=" - nfsd" source="node_exporter.go:104" 2024-09-17T07:18:07.078 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[51349]: time="2024-09-17T07:18:06Z" level=info msg=" - pressure" source="node_exporter.go:104" 2024-09-17T07:18:07.078 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[51349]: time="2024-09-17T07:18:06Z" level=info msg=" - sockstat" source="node_exporter.go:104" 2024-09-17T07:18:07.078 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[51349]: time="2024-09-17T07:18:06Z" level=info msg=" - stat" source="node_exporter.go:104" 2024-09-17T07:18:07.078 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[51349]: time="2024-09-17T07:18:06Z" level=info msg=" - textfile" source="node_exporter.go:104" 2024-09-17T07:18:07.078 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[51349]: time="2024-09-17T07:18:06Z" level=info msg=" - time" source="node_exporter.go:104" 2024-09-17T07:18:07.078 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[51349]: time="2024-09-17T07:18:06Z" level=info msg=" - uname" source="node_exporter.go:104" 2024-09-17T07:18:07.078 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[51349]: time="2024-09-17T07:18:06Z" level=info msg=" - vmstat" source="node_exporter.go:104" 2024-09-17T07:18:07.078 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[51349]: time="2024-09-17T07:18:06Z" level=info msg=" - xfs" source="node_exporter.go:104" 2024-09-17T07:18:07.078 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[51349]: time="2024-09-17T07:18:06Z" level=info msg=" - zfs" source="node_exporter.go:104" 2024-09-17T07:18:07.078 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a[51349]: time="2024-09-17T07:18:06Z" level=info msg="Listening on :9100" source="node_exporter.go:170" 2024-09-17T07:18:07.079 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 bash[51334]: 8a2cde5bd9ed5da1b1583a93a47a04eba639f455650688b7108ef5d73ca66027 2024-09-17T07:18:07.079 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:18:06 smithi044 systemd[1]: Started Ceph node-exporter.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:18:07.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:18:05.064987+0000 mgr.x (mgr.34814) 22 : cluster [DBG] pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 76 KiB/s wr, 5 op/s 2024-09-17T07:18:07.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:18:05.098112+0000 mgr.x (mgr.34814) 23 : cephadm [INF] No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-09-17T07:18:07.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:18:05.098286+0000 mgr.x (mgr.34814) 24 : cephadm [INF] No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-09-17T07:18:07.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:18:05.198532+0000 mgr.x (mgr.34814) 25 : cephadm [INF] Reconfiguring node-exporter.a (dependencies changed)... 2024-09-17T07:18:07.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:18:05.198837+0000 mgr.x (mgr.34814) 26 : cephadm [INF] Reconfiguring daemon node-exporter.a on smithi044 2024-09-17T07:18:07.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:18:05.690884+0000 mon.a (mon.0) 690 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T07:18:08.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:06.899027+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:08.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:06.906806+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:08.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:18:06.908460+0000 mgr.x (mgr.34814) 27 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-09-17T07:18:08.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:18:06.918053+0000 mgr.x (mgr.34814) 28 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi044 2024-09-17T07:18:08.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:06.899027+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:08.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:06.906806+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:08.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:18:06.908460+0000 mgr.x (mgr.34814) 27 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-09-17T07:18:08.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:18:06.918053+0000 mgr.x (mgr.34814) 28 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi044 2024-09-17T07:18:08.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:06.899027+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:08.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:06.906806+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:08.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:18:06.908460+0000 mgr.x (mgr.34814) 27 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-09-17T07:18:08.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:18:06.918053+0000 mgr.x (mgr.34814) 28 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi044 2024-09-17T07:18:09.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:18:07.065883+0000 mgr.x (mgr.34814) 29 : cluster [DBG] pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 896 B/s rd, 25 KiB/s wr, 2 op/s 2024-09-17T07:18:09.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:18:07.065883+0000 mgr.x (mgr.34814) 29 : cluster [DBG] pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 896 B/s rd, 25 KiB/s wr, 2 op/s 2024-09-17T07:18:09.338 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-09-17T07:18:09.363 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:18:07.065883+0000 mgr.x (mgr.34814) 29 : cluster [DBG] pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 896 B/s rd, 25 KiB/s wr, 2 op/s 2024-09-17T07:18:10.049 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T07:18:10.050 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.a smithi044 *:9093,9094 running (4m) 19s ago 4m 19.2M - 0.20.0 0881eb8f169f 42aa02852f04 2024-09-17T07:18:10.050 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.a smithi179 *:3000 running (4m) 18s ago 4m 34.5M - 6.7.4 557c83e11646 2436fcd841b8 2024-09-17T07:18:10.050 INFO:teuthology.orchestra.run.smithi044.stdout:iscsi.foo.smithi044.elmtfk smithi044 error 19s ago 3m - - 2024-09-17T07:18:10.050 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.x smithi179 *:8443 running (31s) 18s ago 8m 463M - 18.2.4-855-g19e2a2be 980a0bfcd684 c9b478bf9874 2024-09-17T07:18:10.050 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.y smithi044 *:9283 running (9m) 19s ago 9m 117M - 16.2.0 24ecd6d5f14c 1b5bd0a0cb11 2024-09-17T07:18:10.051 INFO:teuthology.orchestra.run.smithi044.stdout:mon.a smithi044 running (10m) 19s ago 10m 51.5M 2048M 16.2.0 24ecd6d5f14c 7905efa25a24 2024-09-17T07:18:10.051 INFO:teuthology.orchestra.run.smithi044.stdout:mon.b smithi179 running (8m) 18s ago 8m 45.0M 2048M 16.2.0 24ecd6d5f14c 198f81fe2641 2024-09-17T07:18:10.051 INFO:teuthology.orchestra.run.smithi044.stdout:mon.c smithi044 running (9m) 19s ago 9m 43.0M 2048M 16.2.0 24ecd6d5f14c e0d29cfe87a2 2024-09-17T07:18:10.051 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.a smithi044 *:9100 starting - - - - 2024-09-17T07:18:10.051 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.b smithi179 *:9100 running (5m) 18s ago 5m 15.1M - 0.18.1 e5a616e4b9cf 2106bcb6f0e8 2024-09-17T07:18:10.052 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (8m) 19s ago 8m 45.3M 4096M 16.2.0 24ecd6d5f14c 4517de6029ac 2024-09-17T07:18:10.052 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (8m) 19s ago 8m 44.5M 4096M 16.2.0 24ecd6d5f14c 2d5122cbeff9 2024-09-17T07:18:10.052 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (7m) 19s ago 7m 42.5M 4096M 16.2.0 24ecd6d5f14c 7fd2f30be874 2024-09-17T07:18:10.052 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi044 running (7m) 19s ago 7m 47.5M 4096M 16.2.0 24ecd6d5f14c 2b7d623395a7 2024-09-17T07:18:10.052 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi179 running (7m) 18s ago 7m 50.2M 4096M 16.2.0 24ecd6d5f14c 911e6f47c822 2024-09-17T07:18:10.052 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi179 running (6m) 18s ago 6m 46.4M 4096M 16.2.0 24ecd6d5f14c e56060742dff 2024-09-17T07:18:10.053 INFO:teuthology.orchestra.run.smithi044.stdout:osd.6 smithi179 running (6m) 18s ago 6m 44.0M 4096M 16.2.0 24ecd6d5f14c 7a5c30b68cbc 2024-09-17T07:18:10.053 INFO:teuthology.orchestra.run.smithi044.stdout:osd.7 smithi179 running (6m) 18s ago 6m 49.5M 4096M 16.2.0 24ecd6d5f14c b0d186fbf8a6 2024-09-17T07:18:10.053 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.a smithi179 *:9095 running (4m) 18s ago 5m 36.2M - 2.18.1 de242295e225 3b50713c0562 2024-09-17T07:18:10.053 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi044.nhyhnh smithi044 *:8000 running (3m) 19s ago 3m 36.4M - 16.2.0 24ecd6d5f14c 9b455479614f 2024-09-17T07:18:10.053 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi179.tqlzyy smithi179 *:8000 running (3m) 18s ago 3m 40.1M - 16.2.0 24ecd6d5f14c c8d1693241ce 2024-09-17T07:18:10.053 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi044.vnufiz smithi044 *:80 running (3m) 19s ago 3m 36.6M - 16.2.0 24ecd6d5f14c 6eae2bff5ec8 2024-09-17T07:18:10.054 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi179.ecjhll smithi179 *:80 running (3m) 18s ago 3m 39.9M - 16.2.0 24ecd6d5f14c 253109ee0394 2024-09-17T07:18:10.452 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:18:10.452 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-09-17T07:18:10.452 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 3 2024-09-17T07:18:10.452 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:18:10.452 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-09-17T07:18:10.452 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 1, 2024-09-17T07:18:10.452 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 1 2024-09-17T07:18:10.452 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:18:10.452 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-09-17T07:18:10.452 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2024-09-17T07:18:10.453 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:18:10.453 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": {}, 2024-09-17T07:18:10.453 INFO:teuthology.orchestra.run.smithi044.stdout: "rgw": { 2024-09-17T07:18:10.453 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2024-09-17T07:18:10.453 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:18:10.453 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-09-17T07:18:10.453 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 16, 2024-09-17T07:18:10.453 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 1 2024-09-17T07:18:10.453 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-09-17T07:18:10.454 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:18:10.787 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:18:10.787 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9", 2024-09-17T07:18:10.787 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-09-17T07:18:10.787 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T07:18:10.787 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [], 2024-09-17T07:18:10.787 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "1/23 daemons upgraded", 2024-09-17T07:18:10.787 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "", 2024-09-17T07:18:10.787 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-09-17T07:18:10.787 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:18:11.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:18:09.066482+0000 mgr.x (mgr.34814) 30 : cluster [DBG] pgmap v18: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 896 B/s rd, 25 KiB/s wr, 2 op/s 2024-09-17T07:18:11.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:09.324975+0000 mgr.x (mgr.34814) 31 : audit [DBG] from='client.15033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:18:11.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:09.684553+0000 mgr.x (mgr.34814) 32 : audit [DBG] from='client.15039 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:18:11.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:10.451945+0000 mon.c (mon.1) 34 : audit [DBG] from='client.? 172.21.15.44:0/2616011118' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:18:11.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:18:09.066482+0000 mgr.x (mgr.34814) 30 : cluster [DBG] pgmap v18: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 896 B/s rd, 25 KiB/s wr, 2 op/s 2024-09-17T07:18:11.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:09.324975+0000 mgr.x (mgr.34814) 31 : audit [DBG] from='client.15033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:18:11.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:09.684553+0000 mgr.x (mgr.34814) 32 : audit [DBG] from='client.15039 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:18:11.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:10 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:10.451945+0000 mon.c (mon.1) 34 : audit [DBG] from='client.? 172.21.15.44:0/2616011118' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:18:11.187 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-09-17T07:18:11.187 INFO:teuthology.orchestra.run.smithi044.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-17T07:18:11.187 INFO:teuthology.orchestra.run.smithi044.stdout: daemon iscsi.foo.smithi044.elmtfk on smithi044 is in error state 2024-09-17T07:18:11.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:18:09.066482+0000 mgr.x (mgr.34814) 30 : cluster [DBG] pgmap v18: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 896 B/s rd, 25 KiB/s wr, 2 op/s 2024-09-17T07:18:11.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:09.324975+0000 mgr.x (mgr.34814) 31 : audit [DBG] from='client.15033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:18:11.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:09.684553+0000 mgr.x (mgr.34814) 32 : audit [DBG] from='client.15039 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:18:11.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:10.451945+0000 mon.c (mon.1) 34 : audit [DBG] from='client.? 172.21.15.44:0/2616011118' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:18:12.283 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:10.040100+0000 mgr.x (mgr.34814) 33 : audit [DBG] from='client.15045 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:18:12.284 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:10.786425+0000 mgr.x (mgr.34814) 34 : audit [DBG] from='client.15057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:18:12.284 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:11.187375+0000 mon.a (mon.0) 693 : audit [DBG] from='client.? 172.21.15.44:0/3265761335' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:18:12.284 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:10.040100+0000 mgr.x (mgr.34814) 33 : audit [DBG] from='client.15045 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:18:12.284 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:10.786425+0000 mgr.x (mgr.34814) 34 : audit [DBG] from='client.15057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:18:12.284 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:11.187375+0000 mon.a (mon.0) 693 : audit [DBG] from='client.? 172.21.15.44:0/3265761335' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:18:12.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:11 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:10.040100+0000 mgr.x (mgr.34814) 33 : audit [DBG] from='client.15045 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:18:12.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:11 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:10.786425+0000 mgr.x (mgr.34814) 34 : audit [DBG] from='client.15057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:18:12.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:11 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:11.187375+0000 mon.a (mon.0) 693 : audit [DBG] from='client.? 172.21.15.44:0/3265761335' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:18:13.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:18:11.067269+0000 mgr.x (mgr.34814) 35 : cluster [DBG] pgmap v19: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 896 B/s rd, 25 KiB/s wr, 2 op/s 2024-09-17T07:18:13.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:12 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:18:11.067269+0000 mgr.x (mgr.34814) 35 : cluster [DBG] pgmap v19: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 896 B/s rd, 25 KiB/s wr, 2 op/s 2024-09-17T07:18:13.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:12 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:18:11.067269+0000 mgr.x (mgr.34814) 35 : cluster [DBG] pgmap v19: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 896 B/s rd, 25 KiB/s wr, 2 op/s 2024-09-17T07:18:14.014 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:18:13 smithi044 systemd[1]: Stopping Ceph alertmanager.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:18:14.324 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:18:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a[43179]: level=info ts=2024-09-17T07:18:14.013Z caller=main.go:536 msg="Received SIGTERM, exiting gracefully..." 2024-09-17T07:18:14.324 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:18:14 smithi044 podman[51796]: 2024-09-17 07:18:14.045017852 +0000 UTC m=+0.159857898 container died 42aa02852f042d4d5857f976c8b3c45e5a5949b833fd1dc725707c52e9f978ca (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a, maintainer=The Prometheus Authors ) 2024-09-17T07:18:14.825 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:18:14 smithi044 podman[51796]: 2024-09-17 07:18:14.505310256 +0000 UTC m=+0.620150285 container cleanup 42aa02852f042d4d5857f976c8b3c45e5a5949b833fd1dc725707c52e9f978ca (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a, maintainer=The Prometheus Authors ) 2024-09-17T07:18:14.825 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:18:14 smithi044 podman[51796]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a 2024-09-17T07:18:15.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:14 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:18:13.067777+0000 mgr.x (mgr.34814) 36 : cluster [DBG] pgmap v20: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:18:15.233 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:18:13.067777+0000 mgr.x (mgr.34814) 36 : cluster [DBG] pgmap v20: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:18:15.233 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:18:13.067777+0000 mgr.x (mgr.34814) 36 : cluster [DBG] pgmap v20: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:18:15.234 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:18:14 smithi044 podman[51808]: 2024-09-17 07:18:14.906626036 +0000 UTC m=+0.856110910 container remove 42aa02852f042d4d5857f976c8b3c45e5a5949b833fd1dc725707c52e9f978ca (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a, maintainer=The Prometheus Authors ) 2024-09-17T07:18:15.234 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:18:15 smithi044 podman[51808]: 2024-09-17 07:18:15.040830266 +0000 UTC m=+0.990315137 volume remove 1ecb3ab103e7da8548d62f0456d8038503692201d44ada00a39fba39cc226e77 2024-09-17T07:18:15.234 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:18:15 smithi044 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-17T07:18:15.234 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:18:15 smithi044 systemd[1]: Stopped Ceph alertmanager.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:18:15.234 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:18:15 smithi044 systemd[1]: Starting Ceph alertmanager.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:18:15.563 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:18:15 smithi044 podman[51853]: 2024-09-17 07:18:15.233377738 +0000 UTC m=+0.114432188 volume create 6de02fde8c573995b48b9c1bfe4f2d0631631890a5a56e671a890f125c3dae92 2024-09-17T07:18:15.563 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:18:15 smithi044 podman[51853]: 2024-09-17 07:18:15.137139405 +0000 UTC m=+0.018193837 image pull 0881eb8f169f5556a292b4e2c01d683172b12830a62a9225a98a8e206bb734f0 docker.io/prom/alertmanager:v0.20.0 2024-09-17T07:18:15.563 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:18:15 smithi044 podman[51853]: 2024-09-17 07:18:15.350542832 +0000 UTC m=+0.231597239 container create 0473d4b023adfcada42506350b000e48450c5c701f432209eb0668c0275c9080 (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a, maintainer=The Prometheus Authors ) 2024-09-17T07:18:16.287 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:18:15 smithi044 podman[51853]: 2024-09-17 07:18:15.922885741 +0000 UTC m=+0.803940165 container init 0473d4b023adfcada42506350b000e48450c5c701f432209eb0668c0275c9080 (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a, maintainer=The Prometheus Authors ) 2024-09-17T07:18:16.288 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:18:15 smithi044 podman[51853]: 2024-09-17 07:18:15.927878672 +0000 UTC m=+0.808933085 container start 0473d4b023adfcada42506350b000e48450c5c701f432209eb0668c0275c9080 (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a, maintainer=The Prometheus Authors ) 2024-09-17T07:18:16.288 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:18:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a[51867]: level=info ts=2024-09-17T07:18:15.951Z caller=main.go:231 msg="Starting Alertmanager" version="(version=0.20.0, branch=HEAD, revision=f74be0400a6243d10bb53812d6fa408ad71ff32d)" 2024-09-17T07:18:16.288 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:18:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a[51867]: level=info ts=2024-09-17T07:18:15.951Z caller=main.go:232 build_context="(go=go1.13.5, user=root@00c3106655f8, date=20191211-14:13:14)" 2024-09-17T07:18:16.288 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:18:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a[51867]: level=info ts=2024-09-17T07:18:15.952Z caller=cluster.go:161 component=cluster msg="setting advertise address explicitly" addr=172.21.15.44 port=9094 2024-09-17T07:18:16.288 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:18:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a[51867]: level=info ts=2024-09-17T07:18:15.953Z caller=cluster.go:623 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-17T07:18:16.288 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:18:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a[51867]: level=info ts=2024-09-17T07:18:15.973Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-17T07:18:16.288 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:18:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a[51867]: level=info ts=2024-09-17T07:18:15.974Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-17T07:18:16.288 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:18:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a[51867]: level=info ts=2024-09-17T07:18:15.977Z caller=main.go:497 msg=Listening address=:9093 2024-09-17T07:18:16.288 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:18:16 smithi044 bash[51853]: 0473d4b023adfcada42506350b000e48450c5c701f432209eb0668c0275c9080 2024-09-17T07:18:16.288 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:18:16 smithi044 systemd[1]: Started Ceph alertmanager.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:18:17.231 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:18:15.068264+0000 mgr.x (mgr.34814) 37 : cluster [DBG] pgmap v21: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:18:17.231 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:16.065134+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:17.231 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:16.071079+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:17.231 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:16.072925+0000 mon.b (mon.2) 59 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi044.elmtfk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T07:18:17.231 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:16.073305+0000 mon.a (mon.0) 696 : audit [INF] from='mgr.34814 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi044.elmtfk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T07:18:17.232 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:16.077691+0000 mon.b (mon.2) 60 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:18:17.232 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:18:15.068264+0000 mgr.x (mgr.34814) 37 : cluster [DBG] pgmap v21: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:18:17.232 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:16.065134+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:17.232 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:16.071079+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:17.232 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:16.072925+0000 mon.b (mon.2) 59 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi044.elmtfk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T07:18:17.232 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:16.073305+0000 mon.a (mon.0) 696 : audit [INF] from='mgr.34814 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi044.elmtfk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T07:18:17.232 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:16.077691+0000 mon.b (mon.2) 60 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:18:17.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:16 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:18:15.068264+0000 mgr.x (mgr.34814) 37 : cluster [DBG] pgmap v21: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:18:17.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:16 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:16.065134+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:17.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:16 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:16.071079+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:17.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:16 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:16.072925+0000 mon.b (mon.2) 59 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi044.elmtfk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T07:18:17.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:16 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:16.073305+0000 mon.a (mon.0) 696 : audit [INF] from='mgr.34814 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi044.elmtfk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T07:18:17.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:16 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:16.077691+0000 mon.b (mon.2) 60 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:18:18.215 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:18:17 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a[51867]: level=info ts=2024-09-17T07:18:17.953Z caller=cluster.go:648 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00006492s 2024-09-17T07:18:18.215 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:17 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:18:16.072554+0000 mgr.x (mgr.34814) 38 : cephadm [INF] Reconfiguring iscsi.foo.smithi044.elmtfk (dependencies changed)... 2024-09-17T07:18:18.215 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:17 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:18:16.078568+0000 mgr.x (mgr.34814) 39 : cephadm [INF] Reconfiguring daemon iscsi.foo.smithi044.elmtfk on smithi044 2024-09-17T07:18:18.215 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:17 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:18:16.072554+0000 mgr.x (mgr.34814) 38 : cephadm [INF] Reconfiguring iscsi.foo.smithi044.elmtfk (dependencies changed)... 2024-09-17T07:18:18.216 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:17 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:18:16.078568+0000 mgr.x (mgr.34814) 39 : cephadm [INF] Reconfiguring daemon iscsi.foo.smithi044.elmtfk on smithi044 2024-09-17T07:18:18.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:18:16.072554+0000 mgr.x (mgr.34814) 38 : cephadm [INF] Reconfiguring iscsi.foo.smithi044.elmtfk (dependencies changed)... 2024-09-17T07:18:18.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:18:16.078568+0000 mgr.x (mgr.34814) 39 : cephadm [INF] Reconfiguring daemon iscsi.foo.smithi044.elmtfk on smithi044 2024-09-17T07:18:19.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:18:17.069210+0000 mgr.x (mgr.34814) 40 : cluster [DBG] pgmap v22: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:18:19.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:18.742085+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:19.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:18.748162+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:19.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:18:17.069210+0000 mgr.x (mgr.34814) 40 : cluster [DBG] pgmap v22: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:18:19.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:18.742085+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:19.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:18.748162+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:19.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:18 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:18:17.069210+0000 mgr.x (mgr.34814) 40 : cluster [DBG] pgmap v22: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:18:19.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:18 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:18.742085+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:19.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:18 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:18.748162+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:20.183 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:18:18.750167+0000 mgr.x (mgr.34814) 41 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T07:18:20.184 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:18:18.975988+0000 mgr.x (mgr.34814) 42 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi179 2024-09-17T07:18:20.184 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:19.197525+0000 mon.a (mon.0) 699 : audit [DBG] from='client.? 172.21.15.44:0/4041328085' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:18:20.184 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:19.530198+0000 mon.a (mon.0) 700 : audit [INF] from='client.? 172.21.15.44:0/521493954' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:6801/3175338091"}]: access denied 2024-09-17T07:18:20.184 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:18:18.750167+0000 mgr.x (mgr.34814) 41 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T07:18:20.184 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:18:18.975988+0000 mgr.x (mgr.34814) 42 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi179 2024-09-17T07:18:20.184 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:19.197525+0000 mon.a (mon.0) 699 : audit [DBG] from='client.? 172.21.15.44:0/4041328085' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:18:20.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:19.530198+0000 mon.a (mon.0) 700 : audit [INF] from='client.? 172.21.15.44:0/521493954' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:6801/3175338091"}]: access denied 2024-09-17T07:18:20.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:19 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:18:18.750167+0000 mgr.x (mgr.34814) 41 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T07:18:20.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:19 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:18:18.975988+0000 mgr.x (mgr.34814) 42 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi179 2024-09-17T07:18:20.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:19 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:19.197525+0000 mon.a (mon.0) 699 : audit [DBG] from='client.? 172.21.15.44:0/4041328085' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:18:20.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:19 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:19.530198+0000 mon.a (mon.0) 700 : audit [INF] from='client.? 172.21.15.44:0/521493954' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:6801/3175338091"}]: access denied 2024-09-17T07:18:21.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:18:19.069585+0000 mgr.x (mgr.34814) 43 : cluster [DBG] pgmap v23: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:18:21.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:18:19.069585+0000 mgr.x (mgr.34814) 43 : cluster [DBG] pgmap v23: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:18:21.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:18:19.069585+0000 mgr.x (mgr.34814) 43 : cluster [DBG] pgmap v23: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:18:23.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:22 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:18:21.070209+0000 mgr.x (mgr.34814) 44 : cluster [DBG] pgmap v24: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:23.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:22 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:18:21.070209+0000 mgr.x (mgr.34814) 44 : cluster [DBG] pgmap v24: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:23.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:18:21.070209+0000 mgr.x (mgr.34814) 44 : cluster [DBG] pgmap v24: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:25.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:18:23.070593+0000 mgr.x (mgr.34814) 45 : cluster [DBG] pgmap v25: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:25.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:18:23.070593+0000 mgr.x (mgr.34814) 45 : cluster [DBG] pgmap v25: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:25.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:24 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:18:23.070593+0000 mgr.x (mgr.34814) 45 : cluster [DBG] pgmap v25: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:26.324 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:18:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a[51867]: level=info ts=2024-09-17T07:18:25.954Z caller=cluster.go:640 component=cluster msg="gossip settled; proceeding" elapsed=10.000545325s 2024-09-17T07:18:27.047 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:26 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:18:25.070986+0000 mgr.x (mgr.34814) 46 : cluster [DBG] pgmap v26: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:27.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:18:25.070986+0000 mgr.x (mgr.34814) 46 : cluster [DBG] pgmap v26: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:27.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:18:25.070986+0000 mgr.x (mgr.34814) 46 : cluster [DBG] pgmap v26: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:28.145 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:28 smithi179 systemd[1]: Stopping Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:18:28.528 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:28 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[37172]: level=warn ts=2024-09-17T07:18:28.144Z caller=main.go:524 msg="Received SIGTERM, exiting gracefully..." 2024-09-17T07:18:28.528 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:28 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[37172]: level=info ts=2024-09-17T07:18:28.144Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2024-09-17T07:18:28.528 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:28 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[37172]: level=info ts=2024-09-17T07:18:28.144Z caller=main.go:561 msg="Stopping notify discovery manager..." 2024-09-17T07:18:28.528 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:28 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[37172]: level=info ts=2024-09-17T07:18:28.144Z caller=main.go:583 msg="Stopping scrape manager..." 2024-09-17T07:18:28.528 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:28 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[37172]: level=info ts=2024-09-17T07:18:28.144Z caller=main.go:543 msg="Scrape discovery manager stopped" 2024-09-17T07:18:28.528 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:28 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[37172]: level=info ts=2024-09-17T07:18:28.144Z caller=main.go:557 msg="Notify discovery manager stopped" 2024-09-17T07:18:28.528 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:28 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[37172]: level=info ts=2024-09-17T07:18:28.145Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2024-09-17T07:18:28.528 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:28 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[37172]: level=info ts=2024-09-17T07:18:28.145Z caller=main.go:577 msg="Scrape manager stopped" 2024-09-17T07:18:28.528 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:28 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[37172]: level=info ts=2024-09-17T07:18:28.145Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2024-09-17T07:18:28.529 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:28 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[37172]: level=info ts=2024-09-17T07:18:28.145Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2024-09-17T07:18:28.529 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:28 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[37172]: level=info ts=2024-09-17T07:18:28.145Z caller=main.go:749 msg="Notifier manager stopped" 2024-09-17T07:18:28.529 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:28 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[37172]: level=info ts=2024-09-17T07:18:28.145Z caller=main.go:761 msg="See you next time!" 2024-09-17T07:18:28.529 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:28 smithi179 podman[43485]: 2024-09-17 07:18:28.155163293 +0000 UTC m=+0.114941893 container died 3b50713c05621649d5a83663e1a602c970e0f4bf1e11afbdbce3394dc595a10e (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:18:28.529 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:28 smithi179 podman[43485]: 2024-09-17 07:18:28.527590333 +0000 UTC m=+0.487368941 container cleanup 3b50713c05621649d5a83663e1a602c970e0f4bf1e11afbdbce3394dc595a10e (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:18:28.821 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:28 smithi179 podman[43485]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a 2024-09-17T07:18:29.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:28 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:18:27.071752+0000 mgr.x (mgr.34814) 47 : cluster [DBG] pgmap v27: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:29.114 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:28 smithi179 podman[43497]: 2024-09-17 07:18:28.821053437 +0000 UTC m=+0.663911746 container remove 3b50713c05621649d5a83663e1a602c970e0f4bf1e11afbdbce3394dc595a10e (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:18:29.114 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:28 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-17T07:18:29.114 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:28 smithi179 systemd[1]: Stopped Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:18:29.114 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:28 smithi179 systemd[1]: Starting Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:18:29.114 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 podman[43540]: 2024-09-17 07:18:28.916131452 +0000 UTC m=+0.018122907 image pull de242295e2257c37c8cadfd962369228f8f10b2d48a44259b65fef44ad4f6490 docker.io/prom/prometheus:v2.18.1 2024-09-17T07:18:29.114 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 podman[43540]: 2024-09-17 07:18:29.055386077 +0000 UTC m=+0.157377523 container create 360366ddb35ed3a8676531f5ca9b6f44b03abb11bd1d8aa767deb59896eaef4e (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:18:29.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:28 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:18:27.071752+0000 mgr.x (mgr.34814) 47 : cluster [DBG] pgmap v27: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:29.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:28 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:18:27.071752+0000 mgr.x (mgr.34814) 47 : cluster [DBG] pgmap v27: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:29.614 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 podman[43540]: 2024-09-17 07:18:29.290464023 +0000 UTC m=+0.392455467 container init 360366ddb35ed3a8676531f5ca9b6f44b03abb11bd1d8aa767deb59896eaef4e (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:18:29.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 podman[43540]: 2024-09-17 07:18:29.293223015 +0000 UTC m=+0.395214469 container start 360366ddb35ed3a8676531f5ca9b6f44b03abb11bd1d8aa767deb59896eaef4e (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:18:29.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43555]: level=info ts=2024-09-17T07:18:29.323Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-17T07:18:29.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43555]: level=info ts=2024-09-17T07:18:29.323Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2024-09-17T07:18:29.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43555]: level=info ts=2024-09-17T07:18:29.324Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2024-09-17T07:18:29.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43555]: level=info ts=2024-09-17T07:18:29.324Z caller=main.go:339 host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi179 (none))" 2024-09-17T07:18:29.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43555]: level=info ts=2024-09-17T07:18:29.324Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T07:18:29.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43555]: level=info ts=2024-09-17T07:18:29.324Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T07:18:29.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43555]: level=info ts=2024-09-17T07:18:29.325Z caller=main.go:678 msg="Starting TSDB ..." 2024-09-17T07:18:29.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43555]: level=info ts=2024-09-17T07:18:29.325Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2024-09-17T07:18:29.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43555]: level=info ts=2024-09-17T07:18:29.327Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2024-09-17T07:18:29.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43555]: level=info ts=2024-09-17T07:18:29.327Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-09-17T07:18:29.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43555]: level=info ts=2024-09-17T07:18:29.334Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-09-17T07:18:29.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43555]: level=info ts=2024-09-17T07:18:29.335Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-09-17T07:18:29.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43555]: level=info ts=2024-09-17T07:18:29.335Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=7.698439ms 2024-09-17T07:18:29.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43555]: level=info ts=2024-09-17T07:18:29.336Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2024-09-17T07:18:29.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43555]: level=info ts=2024-09-17T07:18:29.336Z caller=main.go:695 msg="TSDB started" 2024-09-17T07:18:29.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43555]: level=info ts=2024-09-17T07:18:29.336Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T07:18:29.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43555]: level=info ts=2024-09-17T07:18:29.337Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2024-09-17T07:18:29.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43555]: level=info ts=2024-09-17T07:18:29.337Z caller=main.go:561 msg="Stopping notify discovery manager..." 2024-09-17T07:18:29.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43555]: level=info ts=2024-09-17T07:18:29.337Z caller=main.go:583 msg="Stopping scrape manager..." 2024-09-17T07:18:29.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43555]: level=info ts=2024-09-17T07:18:29.337Z caller=main.go:557 msg="Notify discovery manager stopped" 2024-09-17T07:18:29.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43555]: level=info ts=2024-09-17T07:18:29.337Z caller=main.go:543 msg="Scrape discovery manager stopped" 2024-09-17T07:18:29.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43555]: level=info ts=2024-09-17T07:18:29.337Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2024-09-17T07:18:29.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43555]: level=info ts=2024-09-17T07:18:29.337Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2024-09-17T07:18:29.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43555]: level=info ts=2024-09-17T07:18:29.337Z caller=main.go:577 msg="Scrape manager stopped" 2024-09-17T07:18:29.617 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43555]: level=info ts=2024-09-17T07:18:29.337Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2024-09-17T07:18:29.617 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43555]: level=info ts=2024-09-17T07:18:29.337Z caller=main.go:749 msg="Notifier manager stopped" 2024-09-17T07:18:29.617 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43555]: level=error ts=2024-09-17T07:18:29.337Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 14: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain\n line 31: field http_sd_configs not found in type config.plain\n line 45: field http_sd_configs not found in type config.plain\n line 49: field http_sd_configs not found in type config.plain" 2024-09-17T07:18:29.617 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 bash[43540]: 360366ddb35ed3a8676531f5ca9b6f44b03abb11bd1d8aa767deb59896eaef4e 2024-09-17T07:18:29.617 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 systemd[1]: Started Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:18:29.617 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 podman[43573]: 2024-09-17 07:18:29.382917715 +0000 UTC m=+0.037459775 container died 360366ddb35ed3a8676531f5ca9b6f44b03abb11bd1d8aa767deb59896eaef4e (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:18:30.208 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:30 smithi179 systemd[1]: Stopping Ceph node-exporter.b for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:18:30.208 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 podman[43573]: 2024-09-17 07:18:29.952858986 +0000 UTC m=+0.607401027 container remove 360366ddb35ed3a8676531f5ca9b6f44b03abb11bd1d8aa767deb59896eaef4e (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:18:30.208 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2024-09-17T07:18:30.208 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:29 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Failed with result 'exit-code'. 2024-09-17T07:18:30.613 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:30 smithi179 podman[43685]: 2024-09-17 07:18:30.20821483 +0000 UTC m=+0.114476773 container died 2106bcb6f0e8d97c5ee3169986fec7d4e8581f1e77ffd2fce466ea9e2c62ad56 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b, maintainer=The Prometheus Authors ) 2024-09-17T07:18:30.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:30 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:18:29.072192+0000 mgr.x (mgr.34814) 48 : cluster [DBG] pgmap v28: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:30.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:30 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:29.403104+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:30.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:30 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:29.411244+0000 mon.a (mon.0) 702 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:30.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:30 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:18:29.412423+0000 mgr.x (mgr.34814) 49 : cephadm [INF] Reconfiguring node-exporter.b (dependencies changed)... 2024-09-17T07:18:30.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:30 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:18:29.412884+0000 mgr.x (mgr.34814) 50 : cephadm [INF] Reconfiguring daemon node-exporter.b on smithi179 2024-09-17T07:18:30.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:18:29.072192+0000 mgr.x (mgr.34814) 48 : cluster [DBG] pgmap v28: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:30.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:29.403104+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:30.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:29.411244+0000 mon.a (mon.0) 702 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:30.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:18:29.412423+0000 mgr.x (mgr.34814) 49 : cephadm [INF] Reconfiguring node-exporter.b (dependencies changed)... 2024-09-17T07:18:30.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:18:29.412884+0000 mgr.x (mgr.34814) 50 : cephadm [INF] Reconfiguring daemon node-exporter.b on smithi179 2024-09-17T07:18:30.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:18:29.072192+0000 mgr.x (mgr.34814) 48 : cluster [DBG] pgmap v28: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:30.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:29.403104+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:30.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:29.411244+0000 mon.a (mon.0) 702 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:30.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:18:29.412423+0000 mgr.x (mgr.34814) 49 : cephadm [INF] Reconfiguring node-exporter.b (dependencies changed)... 2024-09-17T07:18:30.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:18:29.412884+0000 mgr.x (mgr.34814) 50 : cephadm [INF] Reconfiguring daemon node-exporter.b on smithi179 2024-09-17T07:18:32.225 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 podman[43685]: 2024-09-17 07:18:32.024521198 +0000 UTC m=+1.930783132 container cleanup 2106bcb6f0e8d97c5ee3169986fec7d4e8581f1e77ffd2fce466ea9e2c62ad56 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b, maintainer=The Prometheus Authors ) 2024-09-17T07:18:32.226 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 podman[43685]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b 2024-09-17T07:18:32.541 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:18:31.073060+0000 mgr.x (mgr.34814) 51 : cluster [DBG] pgmap v29: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:32.541 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 podman[43699]: 2024-09-17 07:18:32.225755896 +0000 UTC m=+2.017940924 container remove 2106bcb6f0e8d97c5ee3169986fec7d4e8581f1e77ffd2fce466ea9e2c62ad56 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b, maintainer=The Prometheus Authors ) 2024-09-17T07:18:32.541 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-17T07:18:32.541 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-17T07:18:32.541 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 systemd[1]: Stopped Ceph node-exporter.b for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:18:32.541 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@node-exporter.b.service: Consumed 1.202s CPU time. 2024-09-17T07:18:32.541 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 systemd[1]: Starting Ceph node-exporter.b for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:18:32.542 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 podman[43749]: 2024-09-17 07:18:32.3123495 +0000 UTC m=+0.016715806 image pull e5a616e4b9cf68dfcad7782b78e118be4310022e874d52da85c55923fb615f87 docker.io/prom/node-exporter:v0.18.1 2024-09-17T07:18:32.542 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 podman[43749]: 2024-09-17 07:18:32.410050374 +0000 UTC m=+0.114416681 container create 04c63b5cb5240428d2ca29390cca6860a9d087ba9b24dd18bb83f6957a01844e (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b, maintainer=The Prometheus Authors ) 2024-09-17T07:18:32.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:32 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:18:31.073060+0000 mgr.x (mgr.34814) 51 : cluster [DBG] pgmap v29: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:32.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:32 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:18:31.073060+0000 mgr.x (mgr.34814) 51 : cluster [DBG] pgmap v29: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:32.864 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 podman[43749]: 2024-09-17 07:18:32.620702899 +0000 UTC m=+0.325069208 container init 04c63b5cb5240428d2ca29390cca6860a9d087ba9b24dd18bb83f6957a01844e (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b, maintainer=The Prometheus Authors ) 2024-09-17T07:18:32.864 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 podman[43749]: 2024-09-17 07:18:32.623633533 +0000 UTC m=+0.327999849 container start 04c63b5cb5240428d2ca29390cca6860a9d087ba9b24dd18bb83f6957a01844e (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b, maintainer=The Prometheus Authors ) 2024-09-17T07:18:32.864 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[43765]: time="2024-09-17T07:18:32Z" level=info msg="Starting node_exporter (version=0.18.1, branch=HEAD, revision=3db77732e925c08f675d7404a8c46466b2ece83e)" source="node_exporter.go:156" 2024-09-17T07:18:32.864 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[43765]: time="2024-09-17T07:18:32Z" level=info msg="Build context (go=go1.12.5, user=root@b50852a1acba, date=20190604-16:41:18)" source="node_exporter.go:157" 2024-09-17T07:18:32.865 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[43765]: time="2024-09-17T07:18:32Z" level=info msg="Enabled collectors:" source="node_exporter.go:97" 2024-09-17T07:18:32.865 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[43765]: time="2024-09-17T07:18:32Z" level=info msg=" - arp" source="node_exporter.go:104" 2024-09-17T07:18:32.865 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[43765]: time="2024-09-17T07:18:32Z" level=info msg=" - bcache" source="node_exporter.go:104" 2024-09-17T07:18:32.865 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[43765]: time="2024-09-17T07:18:32Z" level=info msg=" - bonding" source="node_exporter.go:104" 2024-09-17T07:18:32.865 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[43765]: time="2024-09-17T07:18:32Z" level=info msg=" - conntrack" source="node_exporter.go:104" 2024-09-17T07:18:32.865 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[43765]: time="2024-09-17T07:18:32Z" level=info msg=" - cpu" source="node_exporter.go:104" 2024-09-17T07:18:32.865 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[43765]: time="2024-09-17T07:18:32Z" level=info msg=" - cpufreq" source="node_exporter.go:104" 2024-09-17T07:18:32.865 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[43765]: time="2024-09-17T07:18:32Z" level=info msg=" - diskstats" source="node_exporter.go:104" 2024-09-17T07:18:32.865 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[43765]: time="2024-09-17T07:18:32Z" level=info msg=" - edac" source="node_exporter.go:104" 2024-09-17T07:18:32.865 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[43765]: time="2024-09-17T07:18:32Z" level=info msg=" - entropy" source="node_exporter.go:104" 2024-09-17T07:18:32.865 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[43765]: time="2024-09-17T07:18:32Z" level=info msg=" - filefd" source="node_exporter.go:104" 2024-09-17T07:18:32.865 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[43765]: time="2024-09-17T07:18:32Z" level=info msg=" - filesystem" source="node_exporter.go:104" 2024-09-17T07:18:32.866 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[43765]: time="2024-09-17T07:18:32Z" level=info msg=" - hwmon" source="node_exporter.go:104" 2024-09-17T07:18:32.866 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[43765]: time="2024-09-17T07:18:32Z" level=info msg=" - infiniband" source="node_exporter.go:104" 2024-09-17T07:18:32.866 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[43765]: time="2024-09-17T07:18:32Z" level=info msg=" - ipvs" source="node_exporter.go:104" 2024-09-17T07:18:32.866 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[43765]: time="2024-09-17T07:18:32Z" level=info msg=" - loadavg" source="node_exporter.go:104" 2024-09-17T07:18:32.866 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[43765]: time="2024-09-17T07:18:32Z" level=info msg=" - mdadm" source="node_exporter.go:104" 2024-09-17T07:18:32.866 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[43765]: time="2024-09-17T07:18:32Z" level=info msg=" - meminfo" source="node_exporter.go:104" 2024-09-17T07:18:32.866 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[43765]: time="2024-09-17T07:18:32Z" level=info msg=" - netclass" source="node_exporter.go:104" 2024-09-17T07:18:32.866 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[43765]: time="2024-09-17T07:18:32Z" level=info msg=" - netdev" source="node_exporter.go:104" 2024-09-17T07:18:32.866 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[43765]: time="2024-09-17T07:18:32Z" level=info msg=" - netstat" source="node_exporter.go:104" 2024-09-17T07:18:32.866 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[43765]: time="2024-09-17T07:18:32Z" level=info msg=" - nfs" source="node_exporter.go:104" 2024-09-17T07:18:32.866 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[43765]: time="2024-09-17T07:18:32Z" level=info msg=" - nfsd" source="node_exporter.go:104" 2024-09-17T07:18:32.866 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[43765]: time="2024-09-17T07:18:32Z" level=info msg=" - pressure" source="node_exporter.go:104" 2024-09-17T07:18:32.866 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[43765]: time="2024-09-17T07:18:32Z" level=info msg=" - sockstat" source="node_exporter.go:104" 2024-09-17T07:18:32.867 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[43765]: time="2024-09-17T07:18:32Z" level=info msg=" - stat" source="node_exporter.go:104" 2024-09-17T07:18:32.867 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[43765]: time="2024-09-17T07:18:32Z" level=info msg=" - textfile" source="node_exporter.go:104" 2024-09-17T07:18:32.867 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[43765]: time="2024-09-17T07:18:32Z" level=info msg=" - time" source="node_exporter.go:104" 2024-09-17T07:18:32.867 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[43765]: time="2024-09-17T07:18:32Z" level=info msg=" - uname" source="node_exporter.go:104" 2024-09-17T07:18:32.867 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[43765]: time="2024-09-17T07:18:32Z" level=info msg=" - vmstat" source="node_exporter.go:104" 2024-09-17T07:18:32.867 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[43765]: time="2024-09-17T07:18:32Z" level=info msg=" - xfs" source="node_exporter.go:104" 2024-09-17T07:18:32.867 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[43765]: time="2024-09-17T07:18:32Z" level=info msg=" - zfs" source="node_exporter.go:104" 2024-09-17T07:18:32.867 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b[43765]: time="2024-09-17T07:18:32Z" level=info msg="Listening on :9100" source="node_exporter.go:170" 2024-09-17T07:18:32.867 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 bash[43749]: 04c63b5cb5240428d2ca29390cca6860a9d087ba9b24dd18bb83f6957a01844e 2024-09-17T07:18:32.867 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:18:32 smithi179 systemd[1]: Started Ceph node-exporter.b for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:18:34.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:32.749083+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:34.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:32.756815+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:34.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:18:32.758347+0000 mgr.x (mgr.34814) 52 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-09-17T07:18:34.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:18:32.765507+0000 mgr.x (mgr.34814) 53 : cephadm [INF] Regenerating cephadm self-signed grafana TLS certificates 2024-09-17T07:18:34.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:32.832569+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:34.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:32.839416+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:34.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:32.841606+0000 mon.b (mon.2) 61 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T07:18:34.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:32.841956+0000 mgr.x (mgr.34814) 54 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T07:18:34.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:18:32.843887+0000 mgr.x (mgr.34814) 55 : cephadm [INF] Reconfiguring daemon grafana.a on smithi179 2024-09-17T07:18:34.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:32.978489+0000 mon.c (mon.1) 35 : audit [DBG] from='client.? 172.21.15.44:0/2125177724' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:18:34.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:33.315572+0000 mon.c (mon.1) 36 : audit [INF] from='client.? 172.21.15.44:0/1156515245' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:6801/3175338091"}]: access denied 2024-09-17T07:18:34.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:32.749083+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:34.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:32.756815+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:34.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:18:32.758347+0000 mgr.x (mgr.34814) 52 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-09-17T07:18:34.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:18:32.765507+0000 mgr.x (mgr.34814) 53 : cephadm [INF] Regenerating cephadm self-signed grafana TLS certificates 2024-09-17T07:18:34.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:32.832569+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:34.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:32.839416+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:34.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:32.841606+0000 mon.b (mon.2) 61 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T07:18:34.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:32.841956+0000 mgr.x (mgr.34814) 54 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T07:18:34.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:18:32.843887+0000 mgr.x (mgr.34814) 55 : cephadm [INF] Reconfiguring daemon grafana.a on smithi179 2024-09-17T07:18:34.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:32.978489+0000 mon.c (mon.1) 35 : audit [DBG] from='client.? 172.21.15.44:0/2125177724' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:18:34.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:33.315572+0000 mon.c (mon.1) 36 : audit [INF] from='client.? 172.21.15.44:0/1156515245' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:6801/3175338091"}]: access denied 2024-09-17T07:18:34.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:32.749083+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:34.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:32.756815+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:34.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:18:32.758347+0000 mgr.x (mgr.34814) 52 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-09-17T07:18:34.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:18:32.765507+0000 mgr.x (mgr.34814) 53 : cephadm [INF] Regenerating cephadm self-signed grafana TLS certificates 2024-09-17T07:18:34.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:32.832569+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:34.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:32.839416+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:34.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:32.841606+0000 mon.b (mon.2) 61 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T07:18:34.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:32.841956+0000 mgr.x (mgr.34814) 54 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T07:18:34.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:18:32.843887+0000 mgr.x (mgr.34814) 55 : cephadm [INF] Reconfiguring daemon grafana.a on smithi179 2024-09-17T07:18:34.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:32.978489+0000 mon.c (mon.1) 35 : audit [DBG] from='client.? 172.21.15.44:0/2125177724' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:18:34.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:33.315572+0000 mon.c (mon.1) 36 : audit [INF] from='client.? 172.21.15.44:0/1156515245' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:6801/3175338091"}]: access denied 2024-09-17T07:18:34.908 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:18:33.073395+0000 mgr.x (mgr.34814) 56 : cluster [DBG] pgmap v30: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:18:34.908 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:18:33.073395+0000 mgr.x (mgr.34814) 56 : cluster [DBG] pgmap v30: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:18:35.113 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:18:33.073395+0000 mgr.x (mgr.34814) 56 : cluster [DBG] pgmap v30: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:18:37.113 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:18:35.073804+0000 mgr.x (mgr.34814) 57 : cluster [DBG] pgmap v31: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:18:37.324 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:18:35.073804+0000 mgr.x (mgr.34814) 57 : cluster [DBG] pgmap v31: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:18:37.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:36 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:18:35.073804+0000 mgr.x (mgr.34814) 57 : cluster [DBG] pgmap v31: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:18:39.113 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:38 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:18:37.074475+0000 mgr.x (mgr.34814) 58 : cluster [DBG] pgmap v32: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:39.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:18:37.074475+0000 mgr.x (mgr.34814) 58 : cluster [DBG] pgmap v32: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:39.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:18:37.074475+0000 mgr.x (mgr.34814) 58 : cluster [DBG] pgmap v32: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:40.348 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:40 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Scheduled restart job, restart counter is at 1. 2024-09-17T07:18:40.348 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:40 smithi179 systemd[1]: Stopped Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:18:40.348 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:40 smithi179 systemd[1]: Starting Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:18:40.614 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:40 smithi179 podman[43920]: 2024-09-17 07:18:40.357382527 +0000 UTC m=+0.121915758 image pull de242295e2257c37c8cadfd962369228f8f10b2d48a44259b65fef44ad4f6490 docker.io/prom/prometheus:v2.18.1 2024-09-17T07:18:41.114 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:40 smithi179 podman[43920]: 2024-09-17 07:18:40.775056999 +0000 UTC m=+0.539590219 container create b6c9b01dc04762e25dbd81113a63d0f7bf5a38dcf4827da7769bd9780ced9e01 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:18:41.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:40 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:18:39.074854+0000 mgr.x (mgr.34814) 59 : cluster [DBG] pgmap v33: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:41.324 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:18:39.074854+0000 mgr.x (mgr.34814) 59 : cluster [DBG] pgmap v33: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:41.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:18:39.074854+0000 mgr.x (mgr.34814) 59 : cluster [DBG] pgmap v33: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:41.552 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-09-17T07:18:42.202 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T07:18:42.202 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.a smithi044 *:9093,9094 starting - - - - 2024-09-17T07:18:42.202 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.a smithi179 *:3000 running (4m) 50s ago 4m 34.5M - 6.7.4 557c83e11646 2436fcd841b8 2024-09-17T07:18:42.202 INFO:teuthology.orchestra.run.smithi044.stdout:iscsi.foo.smithi044.elmtfk smithi044 starting - - - - 2024-09-17T07:18:42.203 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.x smithi179 *:8443 running (63s) 50s ago 9m 463M - 18.2.4-855-g19e2a2be 980a0bfcd684 c9b478bf9874 2024-09-17T07:18:42.203 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.y smithi044 *:9283 running (10m) 51s ago 10m 117M - 16.2.0 24ecd6d5f14c 1b5bd0a0cb11 2024-09-17T07:18:42.203 INFO:teuthology.orchestra.run.smithi044.stdout:mon.a smithi044 running (10m) 51s ago 10m 51.5M 2048M 16.2.0 24ecd6d5f14c 7905efa25a24 2024-09-17T07:18:42.203 INFO:teuthology.orchestra.run.smithi044.stdout:mon.b smithi179 running (9m) 50s ago 9m 45.0M 2048M 16.2.0 24ecd6d5f14c 198f81fe2641 2024-09-17T07:18:42.203 INFO:teuthology.orchestra.run.smithi044.stdout:mon.c smithi044 running (9m) 51s ago 9m 43.0M 2048M 16.2.0 24ecd6d5f14c e0d29cfe87a2 2024-09-17T07:18:42.203 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.a smithi044 *:9100 starting - - - - 2024-09-17T07:18:42.203 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.b smithi179 *:9100 starting - - - - 2024-09-17T07:18:42.203 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (8m) 51s ago 8m 45.3M 4096M 16.2.0 24ecd6d5f14c 4517de6029ac 2024-09-17T07:18:42.203 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (8m) 51s ago 8m 44.5M 4096M 16.2.0 24ecd6d5f14c 2d5122cbeff9 2024-09-17T07:18:42.203 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (8m) 51s ago 8m 42.5M 4096M 16.2.0 24ecd6d5f14c 7fd2f30be874 2024-09-17T07:18:42.203 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi044 running (7m) 51s ago 7m 47.5M 4096M 16.2.0 24ecd6d5f14c 2b7d623395a7 2024-09-17T07:18:42.203 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi179 running (7m) 50s ago 7m 50.2M 4096M 16.2.0 24ecd6d5f14c 911e6f47c822 2024-09-17T07:18:42.203 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi179 running (7m) 50s ago 7m 46.4M 4096M 16.2.0 24ecd6d5f14c e56060742dff 2024-09-17T07:18:42.203 INFO:teuthology.orchestra.run.smithi044.stdout:osd.6 smithi179 running (6m) 50s ago 6m 44.0M 4096M 16.2.0 24ecd6d5f14c 7a5c30b68cbc 2024-09-17T07:18:42.204 INFO:teuthology.orchestra.run.smithi044.stdout:osd.7 smithi179 running (6m) 50s ago 6m 49.5M 4096M 16.2.0 24ecd6d5f14c b0d186fbf8a6 2024-09-17T07:18:42.204 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.a smithi179 *:9095 starting - - - - 2024-09-17T07:18:42.204 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi044.nhyhnh smithi044 *:8000 running (4m) 51s ago 4m 36.4M - 16.2.0 24ecd6d5f14c 9b455479614f 2024-09-17T07:18:42.204 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi179.tqlzyy smithi179 *:8000 running (4m) 50s ago 4m 40.1M - 16.2.0 24ecd6d5f14c c8d1693241ce 2024-09-17T07:18:42.204 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi044.vnufiz smithi044 *:80 running (4m) 51s ago 4m 36.6M - 16.2.0 24ecd6d5f14c 6eae2bff5ec8 2024-09-17T07:18:42.204 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi179.ecjhll smithi179 *:80 running (4m) 50s ago 4m 39.9M - 16.2.0 24ecd6d5f14c 253109ee0394 2024-09-17T07:18:42.599 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:18:42.600 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-09-17T07:18:42.600 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 3 2024-09-17T07:18:42.600 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:18:42.600 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-09-17T07:18:42.600 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 1, 2024-09-17T07:18:42.600 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 1 2024-09-17T07:18:42.600 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:18:42.600 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-09-17T07:18:42.600 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2024-09-17T07:18:42.600 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:18:42.600 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": {}, 2024-09-17T07:18:42.600 INFO:teuthology.orchestra.run.smithi044.stdout: "rgw": { 2024-09-17T07:18:42.601 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2024-09-17T07:18:42.601 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:18:42.601 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-09-17T07:18:42.601 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 16, 2024-09-17T07:18:42.601 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 1 2024-09-17T07:18:42.601 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-09-17T07:18:42.601 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:18:42.614 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:42 smithi179 podman[43920]: 2024-09-17 07:18:42.342524583 +0000 UTC m=+2.107057800 container init b6c9b01dc04762e25dbd81113a63d0f7bf5a38dcf4827da7769bd9780ced9e01 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:18:42.614 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:42 smithi179 podman[43920]: 2024-09-17 07:18:42.345794373 +0000 UTC m=+2.110327590 container start b6c9b01dc04762e25dbd81113a63d0f7bf5a38dcf4827da7769bd9780ced9e01 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:18:42.614 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43947]: level=info ts=2024-09-17T07:18:42.376Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-17T07:18:42.614 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43947]: level=info ts=2024-09-17T07:18:42.376Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2024-09-17T07:18:42.614 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43947]: level=info ts=2024-09-17T07:18:42.376Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2024-09-17T07:18:42.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43947]: level=info ts=2024-09-17T07:18:42.376Z caller=main.go:339 host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi179 (none))" 2024-09-17T07:18:42.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43947]: level=info ts=2024-09-17T07:18:42.376Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T07:18:42.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43947]: level=info ts=2024-09-17T07:18:42.376Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T07:18:42.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43947]: level=info ts=2024-09-17T07:18:42.377Z caller=main.go:678 msg="Starting TSDB ..." 2024-09-17T07:18:42.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43947]: level=info ts=2024-09-17T07:18:42.377Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2024-09-17T07:18:42.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43947]: level=info ts=2024-09-17T07:18:42.379Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2024-09-17T07:18:42.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43947]: level=info ts=2024-09-17T07:18:42.380Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-09-17T07:18:42.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43947]: level=info ts=2024-09-17T07:18:42.386Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-09-17T07:18:42.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43947]: level=info ts=2024-09-17T07:18:42.387Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-09-17T07:18:42.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43947]: level=info ts=2024-09-17T07:18:42.387Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-09-17T07:18:42.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43947]: level=info ts=2024-09-17T07:18:42.387Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=7.232135ms 2024-09-17T07:18:42.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43947]: level=info ts=2024-09-17T07:18:42.388Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2024-09-17T07:18:42.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43947]: level=info ts=2024-09-17T07:18:42.388Z caller=main.go:695 msg="TSDB started" 2024-09-17T07:18:42.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43947]: level=info ts=2024-09-17T07:18:42.388Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T07:18:42.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43947]: level=info ts=2024-09-17T07:18:42.389Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2024-09-17T07:18:42.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43947]: level=info ts=2024-09-17T07:18:42.389Z caller=main.go:561 msg="Stopping notify discovery manager..." 2024-09-17T07:18:42.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43947]: level=info ts=2024-09-17T07:18:42.389Z caller=main.go:583 msg="Stopping scrape manager..." 2024-09-17T07:18:42.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43947]: level=info ts=2024-09-17T07:18:42.389Z caller=main.go:557 msg="Notify discovery manager stopped" 2024-09-17T07:18:42.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43947]: level=info ts=2024-09-17T07:18:42.389Z caller=main.go:543 msg="Scrape discovery manager stopped" 2024-09-17T07:18:42.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43947]: level=info ts=2024-09-17T07:18:42.389Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2024-09-17T07:18:42.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43947]: level=info ts=2024-09-17T07:18:42.389Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2024-09-17T07:18:42.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43947]: level=info ts=2024-09-17T07:18:42.389Z caller=main.go:577 msg="Scrape manager stopped" 2024-09-17T07:18:42.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43947]: level=info ts=2024-09-17T07:18:42.389Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2024-09-17T07:18:42.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43947]: level=info ts=2024-09-17T07:18:42.389Z caller=main.go:749 msg="Notifier manager stopped" 2024-09-17T07:18:42.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[43947]: level=error ts=2024-09-17T07:18:42.389Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 14: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain\n line 31: field http_sd_configs not found in type config.plain\n line 45: field http_sd_configs not found in type config.plain\n line 49: field http_sd_configs not found in type config.plain" 2024-09-17T07:18:42.617 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:42 smithi179 bash[43920]: b6c9b01dc04762e25dbd81113a63d0f7bf5a38dcf4827da7769bd9780ced9e01 2024-09-17T07:18:42.617 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:42 smithi179 systemd[1]: Started Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:18:42.617 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:42 smithi179 podman[44002]: 2024-09-17 07:18:42.436330011 +0000 UTC m=+0.035638486 container died b6c9b01dc04762e25dbd81113a63d0f7bf5a38dcf4827da7769bd9780ced9e01 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:18:42.940 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:18:42.940 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9", 2024-09-17T07:18:42.940 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-09-17T07:18:42.940 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T07:18:42.940 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [], 2024-09-17T07:18:42.940 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "1/23 daemons upgraded", 2024-09-17T07:18:42.940 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "", 2024-09-17T07:18:42.940 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-09-17T07:18:42.940 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:18:43.114 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:42 smithi179 podman[44002]: 2024-09-17 07:18:42.963661959 +0000 UTC m=+0.562970473 container remove b6c9b01dc04762e25dbd81113a63d0f7bf5a38dcf4827da7769bd9780ced9e01 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:18:43.114 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:42 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2024-09-17T07:18:43.114 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:42 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Failed with result 'exit-code'. 2024-09-17T07:18:43.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:18:41.075576+0000 mgr.x (mgr.34814) 60 : cluster [DBG] pgmap v34: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:43.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:41.538179+0000 mgr.x (mgr.34814) 61 : audit [DBG] from='client.15117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:18:43.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:41.872620+0000 mgr.x (mgr.34814) 62 : audit [DBG] from='client.15123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:18:43.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:42.599748+0000 mon.a (mon.0) 707 : audit [DBG] from='client.? 172.21.15.44:0/2261693617' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:18:43.214 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:18:41.075576+0000 mgr.x (mgr.34814) 60 : cluster [DBG] pgmap v34: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:43.215 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:41.538179+0000 mgr.x (mgr.34814) 61 : audit [DBG] from='client.15117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:18:43.215 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:41.872620+0000 mgr.x (mgr.34814) 62 : audit [DBG] from='client.15123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:18:43.215 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:42.599748+0000 mon.a (mon.0) 707 : audit [DBG] from='client.? 172.21.15.44:0/2261693617' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:18:43.215 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:18:41.075576+0000 mgr.x (mgr.34814) 60 : cluster [DBG] pgmap v34: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:43.215 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:41.538179+0000 mgr.x (mgr.34814) 61 : audit [DBG] from='client.15117 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:18:43.215 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:41.872620+0000 mgr.x (mgr.34814) 62 : audit [DBG] from='client.15123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:18:43.215 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:42.599748+0000 mon.a (mon.0) 707 : audit [DBG] from='client.? 172.21.15.44:0/2261693617' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:18:43.336 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-09-17T07:18:43.337 INFO:teuthology.orchestra.run.smithi044.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-17T07:18:43.337 INFO:teuthology.orchestra.run.smithi044.stdout: daemon iscsi.foo.smithi044.elmtfk on smithi044 is in error state 2024-09-17T07:18:43.754 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:43 smithi179 systemd[1]: Stopping Ceph grafana.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:18:43.754 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:43 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:18:43+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-09-17T07:18:43.754 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:43 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:18:43+0000 lvl=info msg="Stopped NotificationService" logger=server reason="context canceled" 2024-09-17T07:18:43.754 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:43 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:18:43+0000 lvl=info msg="Stopped UsageStatsService" logger=server reason="context canceled" 2024-09-17T07:18:43.755 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:43 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:18:43+0000 lvl=info msg="Stopped AlertEngine" logger=server reason="context canceled" 2024-09-17T07:18:43.755 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:43 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:18:43+0000 lvl=info msg="Stopped CleanUpService" logger=server reason="context canceled" 2024-09-17T07:18:43.755 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:43 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:18:43+0000 lvl=info msg="Stopped provisioningServiceImpl" logger=server reason="context canceled" 2024-09-17T07:18:43.755 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:43 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:18:43+0000 lvl=info msg="Stopped BackendPluginManager" logger=server reason="context canceled" 2024-09-17T07:18:43.755 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:43 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:18:43+0000 lvl=info msg="Stopped RemoteCache" logger=server reason="context canceled" 2024-09-17T07:18:43.755 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:43 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:18:43+0000 lvl=info msg="Stopped UserAuthTokenService" logger=server reason="context canceled" 2024-09-17T07:18:43.755 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:43 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:18:43+0000 lvl=info msg="Stopped PluginManager" logger=server reason="context canceled" 2024-09-17T07:18:43.756 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:43 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:18:43+0000 lvl=info msg="Stopped InternalMetricsService" logger=server reason="context canceled" 2024-09-17T07:18:43.756 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:43 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[36552]: t=2024-09-17T07:18:43+0000 lvl=info msg="Stopped Stream Manager" 2024-09-17T07:18:43.756 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:43 smithi179 podman[44033]: 2024-09-17 07:18:43.563014501 +0000 UTC m=+0.112929045 container died 2436fcd841b8e31657c32d9448424d95e0f4c57548d9c3f03bef30d53eb002f6 (image=docker.io/ceph/ceph-grafana:6.7.4, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a, org.label-schema.schema-version=1.0, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.19.8, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, org.label-schema.vendor=CentOS, description=Ceph Grafana Container, maintainer=Paul Cuzner , org.label-schema.license=GPLv2) 2024-09-17T07:18:44.070 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:43 smithi179 podman[44033]: 2024-09-17 07:18:43.86099876 +0000 UTC m=+0.410913296 container cleanup 2436fcd841b8e31657c32d9448424d95e0f4c57548d9c3f03bef30d53eb002f6 (image=docker.io/ceph/ceph-grafana:6.7.4, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a, summary=Grafana Container configured for Ceph mgr/dashboard integration, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, description=Ceph Grafana Container, io.buildah.version=1.19.8, maintainer=Paul Cuzner , org.label-schema.build-date=20201204, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Base Image) 2024-09-17T07:18:44.071 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:43 smithi179 podman[44033]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a 2024-09-17T07:18:44.071 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:43 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:42.194706+0000 mgr.x (mgr.34814) 63 : audit [DBG] from='client.25048 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:18:44.071 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:43 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:42.939531+0000 mgr.x (mgr.34814) 64 : audit [DBG] from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:18:44.071 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:43 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:43.336994+0000 mon.a (mon.0) 708 : audit [DBG] from='client.? 172.21.15.44:0/3870327276' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:18:44.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:43 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:42.194706+0000 mgr.x (mgr.34814) 63 : audit [DBG] from='client.25048 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:18:44.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:43 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:42.939531+0000 mgr.x (mgr.34814) 64 : audit [DBG] from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:18:44.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:43 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:43.336994+0000 mon.a (mon.0) 708 : audit [DBG] from='client.? 172.21.15.44:0/3870327276' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:18:44.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:43 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:42.194706+0000 mgr.x (mgr.34814) 63 : audit [DBG] from='client.25048 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:18:44.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:43 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:42.939531+0000 mgr.x (mgr.34814) 64 : audit [DBG] from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:18:44.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:43 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:43.336994+0000 mon.a (mon.0) 708 : audit [DBG] from='client.? 172.21.15.44:0/3870327276' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:18:44.333 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 podman[44045]: 2024-09-17 07:18:44.070511344 +0000 UTC m=+0.510016171 container remove 2436fcd841b8e31657c32d9448424d95e0f4c57548d9c3f03bef30d53eb002f6 (image=docker.io/ceph/ceph-grafana:6.7.4, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , org.label-schema.name=CentOS Base Image, org.label-schema.schema-version=1.0, description=Ceph Grafana Container, io.buildah.version=1.19.8, org.label-schema.license=GPLv2, org.label-schema.build-date=20201204, org.label-schema.vendor=CentOS) 2024-09-17T07:18:44.334 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@grafana.a.service: Deactivated successfully. 2024-09-17T07:18:44.334 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 systemd[1]: Stopped Ceph grafana.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:18:44.334 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 systemd[1]: Starting Ceph grafana.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:18:44.334 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 podman[44088]: 2024-09-17 07:18:44.238618172 +0000 UTC m=+0.105211461 container create 43ab0e03dcac946ea54f46ba796dd331f6df77f2fab1eba47fd9b42c27c42180 (image=docker.io/ceph/ceph-grafana:6.7.4, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a, io.buildah.version=1.19.8, org.label-schema.name=CentOS Base Image, org.label-schema.vendor=CentOS, description=Ceph Grafana Container, org.label-schema.license=GPLv2, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, org.label-schema.build-date=20201204, org.label-schema.schema-version=1.0) 2024-09-17T07:18:44.334 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 podman[44088]: 2024-09-17 07:18:44.148420355 +0000 UTC m=+0.015013612 image pull 557c83e11646f123a27b5e4b62ac6c45e7bb8b2e90d6044034d0db5b7019415c docker.io/ceph/ceph-grafana:6.7.4 2024-09-17T07:18:44.614 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 podman[44088]: 2024-09-17 07:18:44.448769254 +0000 UTC m=+0.315362497 container init 43ab0e03dcac946ea54f46ba796dd331f6df77f2fab1eba47fd9b42c27c42180 (image=docker.io/ceph/ceph-grafana:6.7.4, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a, description=Ceph Grafana Container, org.label-schema.vendor=CentOS, summary=Grafana Container configured for Ceph mgr/dashboard integration, org.label-schema.build-date=20201204, io.buildah.version=1.19.8, maintainer=Paul Cuzner , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Base Image) 2024-09-17T07:18:44.614 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 podman[44088]: 2024-09-17 07:18:44.451638992 +0000 UTC m=+0.318232239 container start 43ab0e03dcac946ea54f46ba796dd331f6df77f2fab1eba47fd9b42c27c42180 (image=docker.io/ceph/ceph-grafana:6.7.4, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, org.label-schema.schema-version=1.0, org.label-schema.build-date=20201204, io.buildah.version=1.19.8, maintainer=Paul Cuzner , org.label-schema.name=CentOS Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-17T07:18:44.614 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Starting Grafana" logger=server version=6.7.4 commit=8e44bbc5f5 branch=HEAD compiled=2020-05-26T17:35:38+0000 2024-09-17T07:18:44.615 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-09-17T07:18:44.615 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-09-17T07:18:44.615 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-17T07:18:44.615 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-17T07:18:44.615 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-17T07:18:44.615 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-17T07:18:44.615 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-09-17T07:18:44.615 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-09-17T07:18:44.615 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-09-17T07:18:44.615 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-09-17T07:18:44.615 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-09-17T07:18:44.615 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="App mode production" logger=settings 2024-09-17T07:18:44.615 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Initializing SqlStore" logger=server 2024-09-17T07:18:44.616 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-09-17T07:18:44.616 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Starting DB migration" logger=migrator 2024-09-17T07:18:44.616 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2024-09-17T07:18:44.617 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2024-09-17T07:18:44.617 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2024-09-17T07:18:44.617 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2024-09-17T07:18:44.617 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2024-09-17T07:18:44.617 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2024-09-17T07:18:44.617 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2024-09-17T07:18:44.617 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2024-09-17T07:18:44.617 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2024-09-17T07:18:44.617 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2024-09-17T07:18:44.617 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-09-17T07:18:44.617 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2024-09-17T07:18:44.617 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2024-09-17T07:18:44.617 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2024-09-17T07:18:44.617 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2024-09-17T07:18:44.618 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2024-09-17T07:18:44.618 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2024-09-17T07:18:44.618 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2024-09-17T07:18:44.618 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2024-09-17T07:18:44.618 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2024-09-17T07:18:44.618 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2024-09-17T07:18:44.618 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-09-17T07:18:44.618 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-09-17T07:18:44.618 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-09-17T07:18:44.618 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-09-17T07:18:44.618 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-09-17T07:18:44.618 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-09-17T07:18:44.618 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-09-17T07:18:44.618 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-09-17T07:18:44.619 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-09-17T07:18:44.619 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-09-17T07:18:44.619 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-09-17T07:18:44.619 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-09-17T07:18:44.619 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-09-17T07:18:44.619 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-09-17T07:18:44.619 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-09-17T07:18:44.619 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-09-17T07:18:44.619 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-09-17T07:18:44.619 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-17T07:18:44.619 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-09-17T07:18:44.619 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-09-17T07:18:44.619 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-09-17T07:18:44.620 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-09-17T07:18:44.620 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-09-17T07:18:44.620 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-09-17T07:18:44.620 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-09-17T07:18:44.620 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-09-17T07:18:44.620 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-09-17T07:18:44.620 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-09-17T07:18:44.620 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-09-17T07:18:44.620 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-09-17T07:18:44.620 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-09-17T07:18:44.620 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-09-17T07:18:44.620 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-09-17T07:18:44.620 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-09-17T07:18:44.620 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-09-17T07:18:44.621 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-09-17T07:18:44.621 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-09-17T07:18:44.621 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-09-17T07:18:44.621 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-09-17T07:18:44.621 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-09-17T07:18:44.621 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-09-17T07:18:44.621 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-09-17T07:18:44.621 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-17T07:18:44.621 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-09-17T07:18:44.621 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-17T07:18:44.621 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-09-17T07:18:44.621 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-17T07:18:44.621 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-17T07:18:44.621 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-09-17T07:18:44.622 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-09-17T07:18:44.622 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-09-17T07:18:44.622 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-09-17T07:18:44.622 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-09-17T07:18:44.622 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-09-17T07:18:44.622 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-09-17T07:18:44.622 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-09-17T07:18:44.622 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-09-17T07:18:44.622 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-09-17T07:18:44.622 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-09-17T07:18:44.622 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-09-17T07:18:44.622 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-09-17T07:18:44.622 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-09-17T07:18:44.623 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-09-17T07:18:44.623 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-09-17T07:18:44.623 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-09-17T07:18:44.623 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-09-17T07:18:44.623 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-09-17T07:18:44.623 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-09-17T07:18:44.623 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-09-17T07:18:44.623 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-09-17T07:18:44.623 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-09-17T07:18:44.623 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-09-17T07:18:44.623 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-09-17T07:18:44.623 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-09-17T07:18:44.623 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-09-17T07:18:44.623 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-09-17T07:18:44.624 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-09-17T07:18:44.624 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-09-17T07:18:44.625 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-09-17T07:18:44.625 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-09-17T07:18:44.625 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-09-17T07:18:44.625 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-09-17T07:18:44.625 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-09-17T07:18:44.625 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-09-17T07:18:44.625 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-09-17T07:18:44.625 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-09-17T07:18:44.626 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-09-17T07:18:44.626 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-09-17T07:18:44.626 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-09-17T07:18:44.626 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-17T07:18:44.626 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-17T07:18:44.626 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-09-17T07:18:44.626 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-09-17T07:18:44.626 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-09-17T07:18:44.626 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-09-17T07:18:44.626 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-17T07:18:44.626 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-09-17T07:18:44.626 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-09-17T07:18:44.626 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-17T07:18:44.626 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-09-17T07:18:44.627 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-09-17T07:18:44.627 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-09-17T07:18:44.627 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-09-17T07:18:44.627 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-09-17T07:18:44.627 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-09-17T07:18:44.627 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-09-17T07:18:44.627 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-09-17T07:18:44.627 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-09-17T07:18:44.627 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-09-17T07:18:44.627 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-09-17T07:18:44.627 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-09-17T07:18:44.627 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-09-17T07:18:44.627 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-09-17T07:18:44.627 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-09-17T07:18:44.627 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-09-17T07:18:44.628 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-09-17T07:18:44.628 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-09-17T07:18:44.628 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-09-17T07:18:44.628 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-09-17T07:18:44.628 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-17T07:18:44.628 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-09-17T07:18:44.628 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-09-17T07:18:44.628 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-09-17T07:18:44.628 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-09-17T07:18:44.628 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-09-17T07:18:44.628 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-09-17T07:18:44.628 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-09-17T07:18:44.628 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-09-17T07:18:44.629 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-09-17T07:18:44.629 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-17T07:18:44.629 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-09-17T07:18:44.629 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-09-17T07:18:44.629 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-17T07:18:44.629 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-09-17T07:18:44.629 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-09-17T07:18:44.629 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-09-17T07:18:44.629 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-09-17T07:18:44.629 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-09-17T07:18:44.629 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-09-17T07:18:44.629 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-09-17T07:18:44.629 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-09-17T07:18:44.629 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-09-17T07:18:44.629 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-09-17T07:18:44.630 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-09-17T07:18:44.630 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-09-17T07:18:44.630 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-09-17T07:18:44.630 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-09-17T07:18:44.630 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-09-17T07:18:44.630 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-09-17T07:18:44.630 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-09-17T07:18:44.630 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-17T07:18:44.630 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-09-17T07:18:44.630 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-09-17T07:18:44.630 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-09-17T07:18:44.630 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-09-17T07:18:44.630 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-09-17T07:18:44.630 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-09-17T07:18:44.631 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-09-17T07:18:44.631 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-09-17T07:18:44.631 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-09-17T07:18:44.631 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-09-17T07:18:44.631 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-09-17T07:18:44.631 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-17T07:18:44.631 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-17T07:18:44.631 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-17T07:18:44.631 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-17T07:18:44.631 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-09-17T07:18:44.631 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-09-17T07:18:44.631 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-09-17T07:18:44.631 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-09-17T07:18:44.631 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-17T07:18:44.632 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-09-17T07:18:44.632 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-09-17T07:18:44.632 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-09-17T07:18:44.632 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-09-17T07:18:44.632 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-09-17T07:18:44.632 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-09-17T07:18:44.632 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-09-17T07:18:44.632 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-09-17T07:18:44.632 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-09-17T07:18:44.632 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-09-17T07:18:44.632 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-09-17T07:18:44.632 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-09-17T07:18:44.632 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-09-17T07:18:44.632 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-09-17T07:18:44.633 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-17T07:18:44.633 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-17T07:18:44.633 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-09-17T07:18:44.633 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-09-17T07:18:44.633 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-09-17T07:18:44.633 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-09-17T07:18:44.633 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-09-17T07:18:44.633 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-09-17T07:18:44.633 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-17T07:18:44.633 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-09-17T07:18:44.633 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-09-17T07:18:44.633 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 bash[44088]: 43ab0e03dcac946ea54f46ba796dd331f6df77f2fab1eba47fd9b42c27c42180 2024-09-17T07:18:44.633 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-09-17T07:18:44.633 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-09-17T07:18:44.634 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-09-17T07:18:44.634 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-17T07:18:44.634 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-09-17T07:18:44.634 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-09-17T07:18:44.634 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 systemd[1]: Started Ceph grafana.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:18:44.634 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-09-17T07:18:44.634 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-09-17T07:18:44.634 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-09-17T07:18:44.634 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-09-17T07:18:44.634 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-09-17T07:18:44.634 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-09-17T07:18:44.634 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-09-17T07:18:44.634 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-09-17T07:18:44.634 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-09-17T07:18:44.635 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-09-17T07:18:44.635 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-09-17T07:18:44.635 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-09-17T07:18:44.635 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Initializing HTTPServer" logger=server 2024-09-17T07:18:44.635 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Initializing BackendPluginManager" logger=server 2024-09-17T07:18:44.635 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Initializing PluginManager" logger=server 2024-09-17T07:18:44.635 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Starting plugin search" logger=plugins 2024-09-17T07:18:44.635 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Registering plugin" logger=plugins name="Pie Chart" 2024-09-17T07:18:44.635 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Registering plugin" logger=plugins name="Status Panel" 2024-09-17T07:18:44.635 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Initializing HooksService" logger=server 2024-09-17T07:18:44.635 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Initializing OSSLicensingService" logger=server 2024-09-17T07:18:44.635 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Initializing InternalMetricsService" logger=server 2024-09-17T07:18:44.635 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Initializing RemoteCache" logger=server 2024-09-17T07:18:44.635 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Initializing RenderingService" logger=server 2024-09-17T07:18:44.636 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Initializing AlertEngine" logger=server 2024-09-17T07:18:44.636 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Initializing QuotaService" logger=server 2024-09-17T07:18:44.636 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Initializing ServerLockService" logger=server 2024-09-17T07:18:44.636 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Initializing UserAuthTokenService" logger=server 2024-09-17T07:18:44.636 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Initializing DatasourceCacheService" logger=server 2024-09-17T07:18:44.636 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Initializing LoginService" logger=server 2024-09-17T07:18:44.636 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Initializing SearchService" logger=server 2024-09-17T07:18:44.636 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Initializing TracingService" logger=server 2024-09-17T07:18:44.636 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Initializing UsageStatsService" logger=server 2024-09-17T07:18:44.636 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Initializing CleanUpService" logger=server 2024-09-17T07:18:44.636 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Initializing NotificationService" logger=server 2024-09-17T07:18:44.636 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Initializing provisioningServiceImpl" logger=server 2024-09-17T07:18:44.636 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 2024-09-17T07:18:44.636 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Loki 2024-09-17T07:18:44.637 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Backend rendering via phantomJS" logger=rendering renderer=phantomJS 2024-09-17T07:18:44.637 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=warn msg="phantomJS is deprecated and will be removed in a future release. You should consider migrating from phantomJS to grafana-image-renderer plugin. Read more at https://grafana.com/docs/grafana/latest/administration/image_rendering/" logger=rendering renderer=phantomJS 2024-09-17T07:18:44.637 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="Initializing Stream Manager" 2024-09-17T07:18:44.637 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:18:44+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-09-17T07:18:45.112 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:18:43.076002+0000 mgr.x (mgr.34814) 65 : cluster [DBG] pgmap v35: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:45.113 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:44.575731+0000 mon.a (mon.0) 709 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:45.113 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:44.581485+0000 mon.a (mon.0) 710 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:45.113 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:44.585613+0000 mon.b (mon.2) 62 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T07:18:45.113 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:44.586719+0000 mon.b (mon.2) 63 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi044.front.sepia.ceph.com:9093"}]: dispatch 2024-09-17T07:18:45.113 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:44.593771+0000 mon.a (mon.0) 711 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:45.113 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:44.600690+0000 mon.b (mon.2) 64 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T07:18:45.113 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:44.606534+0000 mon.b (mon.2) 65 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T07:18:45.113 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:44.610823+0000 mon.b (mon.2) 66 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T07:18:45.113 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:44.611909+0000 mon.b (mon.2) 67 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi179.front.sepia.ceph.com:9095"}]: dispatch 2024-09-17T07:18:45.113 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:44.618921+0000 mon.a (mon.0) 712 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:45.113 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:44.629080+0000 mon.b (mon.2) 68 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T07:18:45.113 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:44.630194+0000 mon.b (mon.2) 69 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi179.front.sepia.ceph.com:3000"}]: dispatch 2024-09-17T07:18:45.114 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:44.636833+0000 mon.a (mon.0) 713 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:45.114 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:44.680596+0000 mon.b (mon.2) 70 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:18:45.114 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:44.687803+0000 mon.a (mon.0) 714 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:45.114 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:18:43.076002+0000 mgr.x (mgr.34814) 65 : cluster [DBG] pgmap v35: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:45.114 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:44.575731+0000 mon.a (mon.0) 709 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:45.114 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:44.581485+0000 mon.a (mon.0) 710 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:45.114 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:44.585613+0000 mon.b (mon.2) 62 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T07:18:45.114 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:44.586719+0000 mon.b (mon.2) 63 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi044.front.sepia.ceph.com:9093"}]: dispatch 2024-09-17T07:18:45.114 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:44.593771+0000 mon.a (mon.0) 711 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:45.114 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:44.600690+0000 mon.b (mon.2) 64 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T07:18:45.114 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:44.606534+0000 mon.b (mon.2) 65 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T07:18:45.114 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:44.610823+0000 mon.b (mon.2) 66 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T07:18:45.115 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:44.611909+0000 mon.b (mon.2) 67 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi179.front.sepia.ceph.com:9095"}]: dispatch 2024-09-17T07:18:45.115 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:44.618921+0000 mon.a (mon.0) 712 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:45.115 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:44.629080+0000 mon.b (mon.2) 68 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T07:18:45.115 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:44.630194+0000 mon.b (mon.2) 69 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi179.front.sepia.ceph.com:3000"}]: dispatch 2024-09-17T07:18:45.115 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:44.636833+0000 mon.a (mon.0) 713 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:45.115 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:44.680596+0000 mon.b (mon.2) 70 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:18:45.115 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:44.687803+0000 mon.a (mon.0) 714 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:45.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:18:43.076002+0000 mgr.x (mgr.34814) 65 : cluster [DBG] pgmap v35: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:45.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:44.575731+0000 mon.a (mon.0) 709 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:45.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:44.581485+0000 mon.a (mon.0) 710 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:45.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:44.585613+0000 mon.b (mon.2) 62 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T07:18:45.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:44.586719+0000 mon.b (mon.2) 63 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi044.front.sepia.ceph.com:9093"}]: dispatch 2024-09-17T07:18:45.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:44.593771+0000 mon.a (mon.0) 711 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:45.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:44.600690+0000 mon.b (mon.2) 64 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T07:18:45.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:44.606534+0000 mon.b (mon.2) 65 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T07:18:45.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:44.610823+0000 mon.b (mon.2) 66 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T07:18:45.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:44.611909+0000 mon.b (mon.2) 67 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi179.front.sepia.ceph.com:9095"}]: dispatch 2024-09-17T07:18:45.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:44.618921+0000 mon.a (mon.0) 712 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:45.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:44.629080+0000 mon.b (mon.2) 68 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T07:18:45.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:44.630194+0000 mon.b (mon.2) 69 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi179.front.sepia.ceph.com:3000"}]: dispatch 2024-09-17T07:18:45.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:44.636833+0000 mon.a (mon.0) 713 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:45.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:44.680596+0000 mon.b (mon.2) 70 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:18:45.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:44.687803+0000 mon.a (mon.0) 714 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:46.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:44.585979+0000 mgr.x (mgr.34814) 66 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T07:18:46.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:44.586957+0000 mgr.x (mgr.34814) 67 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi044.front.sepia.ceph.com:9093"}]: dispatch 2024-09-17T07:18:46.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:44.601536+0000 mgr.x (mgr.34814) 68 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T07:18:46.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:44.606882+0000 mgr.x (mgr.34814) 69 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T07:18:46.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:44.611203+0000 mgr.x (mgr.34814) 70 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T07:18:46.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:44.612197+0000 mgr.x (mgr.34814) 71 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi179.front.sepia.ceph.com:9095"}]: dispatch 2024-09-17T07:18:46.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:44.629556+0000 mgr.x (mgr.34814) 72 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T07:18:46.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:44.630450+0000 mgr.x (mgr.34814) 73 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi179.front.sepia.ceph.com:3000"}]: dispatch 2024-09-17T07:18:46.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:18:44.688845+0000 mgr.x (mgr.34814) 74 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T07:18:46.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:18:44.689200+0000 mgr.x (mgr.34814) 75 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T07:18:46.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:44.585979+0000 mgr.x (mgr.34814) 66 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T07:18:46.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:44.586957+0000 mgr.x (mgr.34814) 67 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi044.front.sepia.ceph.com:9093"}]: dispatch 2024-09-17T07:18:46.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:44.601536+0000 mgr.x (mgr.34814) 68 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T07:18:46.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:44.606882+0000 mgr.x (mgr.34814) 69 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T07:18:46.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:44.611203+0000 mgr.x (mgr.34814) 70 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T07:18:46.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:44.612197+0000 mgr.x (mgr.34814) 71 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi179.front.sepia.ceph.com:9095"}]: dispatch 2024-09-17T07:18:46.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:44.629556+0000 mgr.x (mgr.34814) 72 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T07:18:46.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:44.630450+0000 mgr.x (mgr.34814) 73 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi179.front.sepia.ceph.com:3000"}]: dispatch 2024-09-17T07:18:46.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:18:44.688845+0000 mgr.x (mgr.34814) 74 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T07:18:46.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:18:44.689200+0000 mgr.x (mgr.34814) 75 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T07:18:46.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:44.585979+0000 mgr.x (mgr.34814) 66 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T07:18:46.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:44.586957+0000 mgr.x (mgr.34814) 67 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi044.front.sepia.ceph.com:9093"}]: dispatch 2024-09-17T07:18:46.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:44.601536+0000 mgr.x (mgr.34814) 68 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T07:18:46.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:44.606882+0000 mgr.x (mgr.34814) 69 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T07:18:46.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:44.611203+0000 mgr.x (mgr.34814) 70 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T07:18:46.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:44.612197+0000 mgr.x (mgr.34814) 71 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi179.front.sepia.ceph.com:9095"}]: dispatch 2024-09-17T07:18:46.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:44.629556+0000 mgr.x (mgr.34814) 72 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T07:18:46.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:44.630450+0000 mgr.x (mgr.34814) 73 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi179.front.sepia.ceph.com:3000"}]: dispatch 2024-09-17T07:18:46.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:18:44.688845+0000 mgr.x (mgr.34814) 74 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T07:18:46.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:18:44.689200+0000 mgr.x (mgr.34814) 75 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T07:18:47.083 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:18:45.076433+0000 mgr.x (mgr.34814) 76 : cluster [DBG] pgmap v36: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:47.083 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:46.859967+0000 mon.a (mon.0) 715 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:47.083 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:46.861956+0000 mon.b (mon.2) 71 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T07:18:47.083 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:46.862315+0000 mon.a (mon.0) 716 : audit [INF] from='mgr.34814 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T07:18:47.083 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:46.863297+0000 mon.b (mon.2) 72 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T07:18:47.083 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:46.864138+0000 mon.b (mon.2) 73 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:18:47.084 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:18:45.076433+0000 mgr.x (mgr.34814) 76 : cluster [DBG] pgmap v36: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:47.084 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:46.859967+0000 mon.a (mon.0) 715 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:47.084 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:46.861956+0000 mon.b (mon.2) 71 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T07:18:47.084 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:46.862315+0000 mon.a (mon.0) 716 : audit [INF] from='mgr.34814 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T07:18:47.084 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:46.863297+0000 mon.b (mon.2) 72 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T07:18:47.084 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:46.864138+0000 mon.b (mon.2) 73 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:18:47.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:18:45.076433+0000 mgr.x (mgr.34814) 76 : cluster [DBG] pgmap v36: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:47.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:46.859967+0000 mon.a (mon.0) 715 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:47.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:46.861956+0000 mon.b (mon.2) 71 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T07:18:47.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:46.862315+0000 mon.a (mon.0) 716 : audit [INF] from='mgr.34814 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T07:18:47.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:46.863297+0000 mon.b (mon.2) 72 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T07:18:47.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:46.864138+0000 mon.b (mon.2) 73 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:18:48.256 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:18:46.855269+0000 mgr.x (mgr.34814) 77 : cephadm [INF] Upgrade: Updating mgr.y 2024-09-17T07:18:48.256 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:18:46.865008+0000 mgr.x (mgr.34814) 78 : cephadm [INF] Deploying daemon mgr.y on smithi044 2024-09-17T07:18:48.256 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:47.612564+0000 mon.a (mon.0) 717 : audit [DBG] from='client.? 172.21.15.44:0/63249222' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:18:48.257 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:18:46.855269+0000 mgr.x (mgr.34814) 77 : cephadm [INF] Upgrade: Updating mgr.y 2024-09-17T07:18:48.257 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:18:46.865008+0000 mgr.x (mgr.34814) 78 : cephadm [INF] Deploying daemon mgr.y on smithi044 2024-09-17T07:18:48.257 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:47.612564+0000 mon.a (mon.0) 717 : audit [DBG] from='client.? 172.21.15.44:0/63249222' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:18:48.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:18:46.855269+0000 mgr.x (mgr.34814) 77 : cephadm [INF] Upgrade: Updating mgr.y 2024-09-17T07:18:48.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:18:46.865008+0000 mgr.x (mgr.34814) 78 : cephadm [INF] Deploying daemon mgr.y on smithi044 2024-09-17T07:18:48.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:47.612564+0000 mon.a (mon.0) 717 : audit [DBG] from='client.? 172.21.15.44:0/63249222' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:18:48.875 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:48 smithi044 systemd[1]: Stopping Ceph mgr.y for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:18:49.173 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:18:47.077433+0000 mgr.x (mgr.34814) 79 : cluster [DBG] pgmap v37: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:49.174 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:47.949713+0000 mon.c (mon.1) 37 : audit [INF] from='client.? 172.21.15.44:0/1333975050' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:6801/3175338091"}]: access denied 2024-09-17T07:18:49.174 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:18:47.077433+0000 mgr.x (mgr.34814) 79 : cluster [DBG] pgmap v37: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:49.174 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:47.949713+0000 mon.c (mon.1) 37 : audit [INF] from='client.? 172.21.15.44:0/1333975050' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:6801/3175338091"}]: access denied 2024-09-17T07:18:49.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:18:47.077433+0000 mgr.x (mgr.34814) 79 : cluster [DBG] pgmap v37: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:49.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:47.949713+0000 mon.c (mon.1) 37 : audit [INF] from='client.? 172.21.15.44:0/1333975050' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:6801/3175338091"}]: access denied 2024-09-17T07:18:50.354 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:50 smithi044 podman[53164]: 2024-09-17 07:18:50.08500651 +0000 UTC m=+0.178725040 container died 1b5bd0a0cb115ee80ee789f0c3215c98a38d30847177654cf9103e4a409ea40f (image=quay.io/ceph/ceph:v16.2.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.y, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-16.2.0, GIT_BRANCH=HEAD, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, maintainer=Dimitri Savineau , org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-09-17T07:18:50.729 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:50 smithi044 podman[53164]: 2024-09-17 07:18:50.478064719 +0000 UTC m=+0.571783248 container cleanup 1b5bd0a0cb115ee80ee789f0c3215c98a38d30847177654cf9103e4a409ea40f (image=quay.io/ceph/ceph:v16.2.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.y, org.label-schema.license=GPLv2, RELEASE=HEAD, maintainer=Dimitri Savineau , org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, CEPH_POINT_RELEASE=-16.2.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, ceph=True) 2024-09-17T07:18:50.729 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:50 smithi044 bash[53164]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.y 2024-09-17T07:18:51.006 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:50 smithi044 podman[53179]: 2024-09-17 07:18:50.729478048 +0000 UTC m=+0.648022824 container remove 1b5bd0a0cb115ee80ee789f0c3215c98a38d30847177654cf9103e4a409ea40f (image=quay.io/ceph/ceph:v16.2.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.y, org.label-schema.name=CentOS Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=HEAD, org.label-schema.build-date=20201204, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-16.2.0, GIT_CLEAN=True, maintainer=Dimitri Savineau , GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557) 2024-09-17T07:18:51.007 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:50 smithi044 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-09-17T07:18:51.007 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:18:49.078035+0000 mgr.x (mgr.34814) 80 : cluster [DBG] pgmap v38: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:18:51.007 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:18:49.078035+0000 mgr.x (mgr.34814) 80 : cluster [DBG] pgmap v38: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:18:51.298 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:51 smithi044 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@mgr.y.service: Failed with result 'exit-code'. 2024-09-17T07:18:51.298 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:51 smithi044 systemd[1]: Stopped Ceph mgr.y for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:18:51.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:18:49.078035+0000 mgr.x (mgr.34814) 80 : cluster [DBG] pgmap v38: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:18:51.574 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:51 smithi044 systemd[1]: Starting Ceph mgr.y for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:18:51.575 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:51 smithi044 podman[53317]: 2024-09-17 07:18:51.414692444 +0000 UTC m=+0.016503878 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:18:51.575 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:51 smithi044 podman[53317]: 2024-09-17 07:18:51.531099414 +0000 UTC m=+0.132910834 container create 8080dd318a1ca13d3238ec5b7d414656c8924e3f6cc004ec6336b373a0e78f49 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2) 2024-09-17T07:18:51.913 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:51 smithi044 podman[53317]: 2024-09-17 07:18:51.784122307 +0000 UTC m=+0.385933728 container init 8080dd318a1ca13d3238ec5b7d414656c8924e3f6cc004ec6336b373a0e78f49 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y, RELEASE=reef-19e2a2b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-17T07:18:51.913 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:51 smithi044 podman[53317]: 2024-09-17 07:18:51.787326134 +0000 UTC m=+0.389137561 container start 8080dd318a1ca13d3238ec5b7d414656c8924e3f6cc004ec6336b373a0e78f49 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-17T07:18:51.913 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:51 smithi044 ceph-mgr[53336]: -- 172.21.15.44:0/3514769304 <== mon.0 v2:172.21.15.44:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5568616bad00 con 0x55686168f000 2024-09-17T07:18:51.914 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:51 smithi044 bash[53317]: 8080dd318a1ca13d3238ec5b7d414656c8924e3f6cc004ec6336b373a0e78f49 2024-09-17T07:18:51.914 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:51 smithi044 systemd[1]: Started Ceph mgr.y for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:18:52.210 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:18:51.911+0000 7fb7b514f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T07:18:52.210 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:18:51.974+0000 7fb7b514f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T07:18:52.210 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:18:52.135+0000 7fb7b514f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T07:18:52.210 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:18:52.209+0000 7fb7b514f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T07:18:52.523 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:18:52.365+0000 7fb7b514f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T07:18:52.524 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:18:52.522+0000 7fb7b514f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T07:18:53.133 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:18:52.865+0000 7fb7b514f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T07:18:53.133 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:18:52.932+0000 7fb7b514f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T07:18:53.133 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:53 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:18:53.132+0000 7fb7b514f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T07:18:53.133 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:18:51.078734+0000 mgr.x (mgr.34814) 81 : cluster [DBG] pgmap v39: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:53.133 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:51.949101+0000 mon.a (mon.0) 718 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:53.134 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:51.957417+0000 mon.a (mon.0) 719 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:53.134 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:51.961001+0000 mon.b (mon.2) 74 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:18:53.134 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:52.205381+0000 mon.a (mon.0) 720 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:53.134 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:52.210594+0000 mon.a (mon.0) 721 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:53.134 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:18:51.078734+0000 mgr.x (mgr.34814) 81 : cluster [DBG] pgmap v39: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:53.134 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:51.949101+0000 mon.a (mon.0) 718 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:53.134 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:51.957417+0000 mon.a (mon.0) 719 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:53.134 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:51.961001+0000 mon.b (mon.2) 74 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:18:53.134 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:52.205381+0000 mon.a (mon.0) 720 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:53.134 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:52.210594+0000 mon.a (mon.0) 721 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:53.229 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Scheduled restart job, restart counter is at 2. 2024-09-17T07:18:53.229 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 systemd[1]: Stopped Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:18:53.229 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 systemd[1]: Starting Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:18:53.229 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 podman[44351]: 2024-09-17 07:18:53.105894549 +0000 UTC m=+0.017873980 image pull de242295e2257c37c8cadfd962369228f8f10b2d48a44259b65fef44ad4f6490 docker.io/prom/prometheus:v2.18.1 2024-09-17T07:18:53.229 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 podman[44351]: 2024-09-17 07:18:53.229186963 +0000 UTC m=+0.141166393 container create 0d6b491560a8d5f39b196e2364415f8c4dd22ab8d6f7908c6f3c8c720cda4e2a (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:18:53.230 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:18:51.078734+0000 mgr.x (mgr.34814) 81 : cluster [DBG] pgmap v39: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:53.230 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:51.949101+0000 mon.a (mon.0) 718 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:53.230 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:51.957417+0000 mon.a (mon.0) 719 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:53.230 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:51.961001+0000 mon.b (mon.2) 74 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:18:53.230 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:52.205381+0000 mon.a (mon.0) 720 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:53.230 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:52.210594+0000 mon.a (mon.0) 721 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:53.504 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 podman[44351]: 2024-09-17 07:18:53.472926321 +0000 UTC m=+0.384905752 container init 0d6b491560a8d5f39b196e2364415f8c4dd22ab8d6f7908c6f3c8c720cda4e2a (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:18:53.505 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 podman[44351]: 2024-09-17 07:18:53.475693934 +0000 UTC m=+0.387673364 container start 0d6b491560a8d5f39b196e2364415f8c4dd22ab8d6f7908c6f3c8c720cda4e2a (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:18:53.865 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[44367]: level=info ts=2024-09-17T07:18:53.504Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-17T07:18:53.865 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[44367]: level=info ts=2024-09-17T07:18:53.504Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2024-09-17T07:18:53.865 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[44367]: level=info ts=2024-09-17T07:18:53.504Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2024-09-17T07:18:53.866 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[44367]: level=info ts=2024-09-17T07:18:53.504Z caller=main.go:339 host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi179 (none))" 2024-09-17T07:18:53.866 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[44367]: level=info ts=2024-09-17T07:18:53.504Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T07:18:53.866 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[44367]: level=info ts=2024-09-17T07:18:53.504Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T07:18:53.866 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[44367]: level=info ts=2024-09-17T07:18:53.506Z caller=main.go:678 msg="Starting TSDB ..." 2024-09-17T07:18:53.866 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[44367]: level=info ts=2024-09-17T07:18:53.506Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2024-09-17T07:18:53.866 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[44367]: level=info ts=2024-09-17T07:18:53.508Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2024-09-17T07:18:53.867 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[44367]: level=info ts=2024-09-17T07:18:53.509Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=4 2024-09-17T07:18:53.867 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[44367]: level=info ts=2024-09-17T07:18:53.515Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=4 2024-09-17T07:18:53.867 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[44367]: level=info ts=2024-09-17T07:18:53.515Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=4 2024-09-17T07:18:53.867 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[44367]: level=info ts=2024-09-17T07:18:53.516Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=4 2024-09-17T07:18:53.867 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[44367]: level=info ts=2024-09-17T07:18:53.516Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=4 2024-09-17T07:18:53.868 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[44367]: level=info ts=2024-09-17T07:18:53.516Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=7.977675ms 2024-09-17T07:18:53.868 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[44367]: level=info ts=2024-09-17T07:18:53.517Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2024-09-17T07:18:53.868 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[44367]: level=info ts=2024-09-17T07:18:53.517Z caller=main.go:695 msg="TSDB started" 2024-09-17T07:18:53.868 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[44367]: level=info ts=2024-09-17T07:18:53.517Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T07:18:53.868 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[44367]: level=info ts=2024-09-17T07:18:53.518Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2024-09-17T07:18:53.868 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[44367]: level=info ts=2024-09-17T07:18:53.518Z caller=main.go:561 msg="Stopping notify discovery manager..." 2024-09-17T07:18:53.869 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[44367]: level=info ts=2024-09-17T07:18:53.518Z caller=main.go:583 msg="Stopping scrape manager..." 2024-09-17T07:18:53.869 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[44367]: level=info ts=2024-09-17T07:18:53.518Z caller=main.go:557 msg="Notify discovery manager stopped" 2024-09-17T07:18:53.869 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[44367]: level=info ts=2024-09-17T07:18:53.518Z caller=main.go:543 msg="Scrape discovery manager stopped" 2024-09-17T07:18:53.869 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[44367]: level=info ts=2024-09-17T07:18:53.518Z caller=main.go:577 msg="Scrape manager stopped" 2024-09-17T07:18:53.869 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[44367]: level=info ts=2024-09-17T07:18:53.518Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2024-09-17T07:18:53.869 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[44367]: level=info ts=2024-09-17T07:18:53.518Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2024-09-17T07:18:53.870 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[44367]: level=info ts=2024-09-17T07:18:53.518Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2024-09-17T07:18:53.870 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[44367]: level=info ts=2024-09-17T07:18:53.518Z caller=main.go:749 msg="Notifier manager stopped" 2024-09-17T07:18:53.870 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[44367]: level=error ts=2024-09-17T07:18:53.518Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 14: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain\n line 31: field http_sd_configs not found in type config.plain\n line 45: field http_sd_configs not found in type config.plain\n line 49: field http_sd_configs not found in type config.plain" 2024-09-17T07:18:53.870 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 bash[44351]: 0d6b491560a8d5f39b196e2364415f8c4dd22ab8d6f7908c6f3c8c720cda4e2a 2024-09-17T07:18:53.870 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:53 smithi179 systemd[1]: Started Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:18:54.574 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:18:54.080+0000 7fb7b514f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T07:18:54.574 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:18:54.153+0000 7fb7b514f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T07:18:54.614 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:54 smithi179 podman[44384]: 2024-09-17 07:18:54.31879877 +0000 UTC m=+0.792955311 container remove 0d6b491560a8d5f39b196e2364415f8c4dd22ab8d6f7908c6f3c8c720cda4e2a (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:18:54.614 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:54 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2024-09-17T07:18:54.614 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:18:54 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Failed with result 'exit-code'. 2024-09-17T07:18:55.256 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:18:53.079216+0000 mgr.x (mgr.34814) 82 : cluster [DBG] pgmap v40: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:55.256 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:18:53.079216+0000 mgr.x (mgr.34814) 82 : cluster [DBG] pgmap v40: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:55.257 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:18:54.983+0000 7fb7b514f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T07:18:55.257 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:18:55.107+0000 7fb7b514f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T07:18:55.257 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:18:55.255+0000 7fb7b514f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T07:18:55.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:18:53.079216+0000 mgr.x (mgr.34814) 82 : cluster [DBG] pgmap v40: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:55.545 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:18:55.322+0000 7fb7b514f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T07:18:55.545 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:18:55.477+0000 7fb7b514f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T07:18:55.824 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:18:55.548+0000 7fb7b514f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T07:18:55.824 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:18:55.611+0000 7fb7b514f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T07:18:55.825 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:18:55.776+0000 7fb7b514f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T07:18:56.086 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:18:55.858+0000 7fb7b514f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T07:18:56.086 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:55 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:18:55.932+0000 7fb7b514f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T07:18:56.087 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:18:56.023+0000 7fb7b514f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T07:18:56.087 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:18:56.085+0000 7fb7b514f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T07:18:56.340 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:18:56.148+0000 7fb7b514f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T07:18:57.219 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:56 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:18:55.079588+0000 mgr.x (mgr.34814) 83 : cluster [DBG] pgmap v41: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:57.324 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:18:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:18:57.090+0000 7fb7b514f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T07:18:57.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:18:55.079588+0000 mgr.x (mgr.34814) 83 : cluster [DBG] pgmap v41: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:57.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:18:55.079588+0000 mgr.x (mgr.34814) 83 : cluster [DBG] pgmap v41: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:58.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:18:57.093308+0000 mon.a (mon.0) 722 : cluster [DBG] Standby manager daemon y restarted 2024-09-17T07:18:58.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:18:57.093463+0000 mon.a (mon.0) 723 : cluster [DBG] Standby manager daemon y started 2024-09-17T07:18:58.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:57.094333+0000 mon.c (mon.1) 38 : audit [DBG] from='mgr.? 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-17T07:18:58.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:57.094722+0000 mon.c (mon.1) 39 : audit [DBG] from='mgr.? 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T07:18:58.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:57.095449+0000 mon.c (mon.1) 40 : audit [DBG] from='mgr.? 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-17T07:18:58.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:57.095743+0000 mon.c (mon.1) 41 : audit [DBG] from='mgr.? 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T07:18:58.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:57.261606+0000 mon.a (mon.0) 724 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:58.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:57.266752+0000 mon.a (mon.0) 725 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:58.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:57.780548+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:58.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:57.786089+0000 mon.a (mon.0) 727 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:58.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:18:57.093308+0000 mon.a (mon.0) 722 : cluster [DBG] Standby manager daemon y restarted 2024-09-17T07:18:58.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:18:57.093463+0000 mon.a (mon.0) 723 : cluster [DBG] Standby manager daemon y started 2024-09-17T07:18:58.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:57.094333+0000 mon.c (mon.1) 38 : audit [DBG] from='mgr.? 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-17T07:18:58.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:57.094722+0000 mon.c (mon.1) 39 : audit [DBG] from='mgr.? 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T07:18:58.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:57.095449+0000 mon.c (mon.1) 40 : audit [DBG] from='mgr.? 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-17T07:18:58.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:57.095743+0000 mon.c (mon.1) 41 : audit [DBG] from='mgr.? 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T07:18:58.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:57.261606+0000 mon.a (mon.0) 724 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:58.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:57.266752+0000 mon.a (mon.0) 725 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:58.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:57.780548+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:58.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:57.786089+0000 mon.a (mon.0) 727 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:58.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:18:57.093308+0000 mon.a (mon.0) 722 : cluster [DBG] Standby manager daemon y restarted 2024-09-17T07:18:58.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:18:57.093463+0000 mon.a (mon.0) 723 : cluster [DBG] Standby manager daemon y started 2024-09-17T07:18:58.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:57.094333+0000 mon.c (mon.1) 38 : audit [DBG] from='mgr.? 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-17T07:18:58.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:57.094722+0000 mon.c (mon.1) 39 : audit [DBG] from='mgr.? 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T07:18:58.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:57.095449+0000 mon.c (mon.1) 40 : audit [DBG] from='mgr.? 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-17T07:18:58.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:57.095743+0000 mon.c (mon.1) 41 : audit [DBG] from='mgr.? 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T07:18:58.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:57.261606+0000 mon.a (mon.0) 724 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:58.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:57.266752+0000 mon.a (mon.0) 725 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:58.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:57.780548+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:58.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:57.786089+0000 mon.a (mon.0) 727 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:59.275 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:18:57.080353+0000 mgr.x (mgr.34814) 84 : cluster [DBG] pgmap v42: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:59.275 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:18:57.969933+0000 mon.a (mon.0) 728 : cluster [DBG] mgrmap e19: x(active, since 73s), standbys: y 2024-09-17T07:18:59.275 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:58.070243+0000 mon.a (mon.0) 729 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:59.275 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:18:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:18:58.077861+0000 mon.a (mon.0) 730 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:59.276 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:18:57.080353+0000 mgr.x (mgr.34814) 84 : cluster [DBG] pgmap v42: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:59.276 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:18:57.969933+0000 mon.a (mon.0) 728 : cluster [DBG] mgrmap e19: x(active, since 73s), standbys: y 2024-09-17T07:18:59.276 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:58.070243+0000 mon.a (mon.0) 729 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:59.276 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:18:58 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:18:58.077861+0000 mon.a (mon.0) 730 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:59.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:58 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:18:57.080353+0000 mgr.x (mgr.34814) 84 : cluster [DBG] pgmap v42: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:18:59.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:58 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:18:57.969933+0000 mon.a (mon.0) 728 : cluster [DBG] mgrmap e19: x(active, since 73s), standbys: y 2024-09-17T07:18:59.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:58 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:58.070243+0000 mon.a (mon.0) 729 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:18:59.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:18:58 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:18:58.077861+0000 mon.a (mon.0) 730 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:19:01.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:18:59.080779+0000 mgr.x (mgr.34814) 85 : cluster [DBG] pgmap v43: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:19:01.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:18:59.080779+0000 mgr.x (mgr.34814) 85 : cluster [DBG] pgmap v43: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:19:01.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:00 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:18:59.080779+0000 mgr.x (mgr.34814) 85 : cluster [DBG] pgmap v43: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:19:02.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:01.356736+0000 mon.b (mon.2) 75 : audit [DBG] from='client.? 172.21.15.44:0/2002187948' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:19:02.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:01.636848+0000 mon.a (mon.0) 731 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:19:02.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:01.643535+0000 mon.a (mon.0) 732 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:19:02.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:01.645105+0000 mon.b (mon.2) 76 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:19:02.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:01.646005+0000 mon.b (mon.2) 77 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:19:02.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:01.651876+0000 mon.a (mon.0) 733 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:19:02.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:01.696377+0000 mon.a (mon.0) 734 : audit [INF] from='client.? 172.21.15.44:0/152933900' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:6801/3175338091"}]: access denied 2024-09-17T07:19:02.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:01.768999+0000 mon.b (mon.2) 78 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:19:02.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:01.773654+0000 mon.b (mon.2) 79 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-17T07:19:02.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:01.773895+0000 mon.a (mon.0) 735 : audit [INF] from='mgr.34814 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-17T07:19:02.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:01.777979+0000 mon.a (mon.0) 736 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-09-17T07:19:02.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:01.356736+0000 mon.b (mon.2) 75 : audit [DBG] from='client.? 172.21.15.44:0/2002187948' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:19:02.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:01.636848+0000 mon.a (mon.0) 731 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:19:02.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:01.643535+0000 mon.a (mon.0) 732 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:19:02.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:01.645105+0000 mon.b (mon.2) 76 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:19:02.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:01.646005+0000 mon.b (mon.2) 77 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:19:02.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:01.651876+0000 mon.a (mon.0) 733 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:19:02.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:01.696377+0000 mon.a (mon.0) 734 : audit [INF] from='client.? 172.21.15.44:0/152933900' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:6801/3175338091"}]: access denied 2024-09-17T07:19:02.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:01.768999+0000 mon.b (mon.2) 78 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:19:02.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:01.773654+0000 mon.b (mon.2) 79 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-17T07:19:02.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:01.773895+0000 mon.a (mon.0) 735 : audit [INF] from='mgr.34814 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-17T07:19:02.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:01.777979+0000 mon.a (mon.0) 736 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-09-17T07:19:02.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:01 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:01.356736+0000 mon.b (mon.2) 75 : audit [DBG] from='client.? 172.21.15.44:0/2002187948' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:19:02.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:01 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:01.636848+0000 mon.a (mon.0) 731 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:19:02.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:01 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:01.643535+0000 mon.a (mon.0) 732 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:19:02.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:01 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:01.645105+0000 mon.b (mon.2) 76 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:19:02.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:01 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:01.646005+0000 mon.b (mon.2) 77 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:19:02.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:01 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:01.651876+0000 mon.a (mon.0) 733 : audit [INF] from='mgr.34814 ' entity='mgr.x' 2024-09-17T07:19:02.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:01 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:01.696377+0000 mon.a (mon.0) 734 : audit [INF] from='client.? 172.21.15.44:0/152933900' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:6801/3175338091"}]: access denied 2024-09-17T07:19:02.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:01 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:01.768999+0000 mon.b (mon.2) 78 : audit [DBG] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:19:02.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:01 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:01.773654+0000 mon.b (mon.2) 79 : audit [INF] from='mgr.34814 172.21.15.179:0/1520757824' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-17T07:19:02.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:01 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:01.773895+0000 mon.a (mon.0) 735 : audit [INF] from='mgr.34814 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-17T07:19:02.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:01 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:01.777979+0000 mon.a (mon.0) 736 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-09-17T07:19:02.975 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:19:02.660+0000 7f1feb6a7640 -1 mgr handle_mgr_map I was active but no longer am 2024-09-17T07:19:02.975 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: ignoring --setuser ceph since I am not root 2024-09-17T07:19:02.975 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: ignoring --setgroup ceph since I am not root 2024-09-17T07:19:02.975 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:19:02.710+0000 7fa479c01640 1 -- 172.21.15.179:0/510446094 <== mon.2 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557e92088d00 con 0x557e9205c800 2024-09-17T07:19:02.975 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:19:02.815+0000 7fa47c45c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T07:19:02.975 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:19:02.877+0000 7fa47c45c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T07:19:02.976 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:01.081530+0000 mgr.x (mgr.34814) 86 : cluster [DBG] pgmap v44: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-17T07:19:02.976 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:19:01.646753+0000 mgr.x (mgr.34814) 87 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:19:02.976 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:01.647336+0000 mgr.x (mgr.34814) 88 : cluster [DBG] pgmap v45: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2024-09-17T07:19:02.976 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:01.647546+0000 mgr.x (mgr.34814) 89 : cluster [DBG] pgmap v46: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 119 B/s rd, 0 op/s 2024-09-17T07:19:02.976 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:19:01.770982+0000 mgr.x (mgr.34814) 90 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T07:19:02.977 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:19:01.771460+0000 mgr.x (mgr.34814) 91 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T07:19:02.977 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:19:01.773419+0000 mgr.x (mgr.34814) 92 : cephadm [INF] Failing over to other MGR 2024-09-17T07:19:02.977 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:01.968265+0000 mon.a (mon.0) 737 : cluster [WRN] Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T07:19:02.977 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:02.034371+0000 mon.a (mon.0) 738 : cluster [DBG] Standby manager daemon x started 2024-09-17T07:19:02.977 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:02.658601+0000 mon.a (mon.0) 739 : audit [INF] from='mgr.34814 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-17T07:19:02.977 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:02.658664+0000 mon.a (mon.0) 740 : cluster [DBG] mgrmap e20: y(active, starting, since 0.884112s), standbys: x 2024-09-17T07:19:02.977 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:02.661369+0000 mon.c (mon.1) 42 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:19:02.977 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:02.661548+0000 mon.c (mon.1) 43 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:19:02.977 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:02.661714+0000 mon.c (mon.1) 44 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:19:02.977 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:02.663112+0000 mon.c (mon.1) 45 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T07:19:02.977 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:02.663320+0000 mon.c (mon.1) 46 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T07:19:02.977 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:02.663589+0000 mon.c (mon.1) 47 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T07:19:02.977 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:02.663911+0000 mon.c (mon.1) 48 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T07:19:02.977 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:02.664252+0000 mon.c (mon.1) 49 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:19:02.978 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:02.664500+0000 mon.c (mon.1) 50 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:19:02.978 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:02.664699+0000 mon.c (mon.1) 51 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:19:02.978 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:02.665015+0000 mon.c (mon.1) 52 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:19:02.978 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:02.665385+0000 mon.c (mon.1) 53 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:19:02.978 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:02.665671+0000 mon.c (mon.1) 54 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:19:02.978 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:02.665978+0000 mon.c (mon.1) 55 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T07:19:02.978 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:02.666155+0000 mon.c (mon.1) 56 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T07:19:02.978 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:02.666579+0000 mon.c (mon.1) 57 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T07:19:02.978 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:02.798834+0000 mon.a (mon.0) 741 : cluster [INF] Manager daemon y is now available 2024-09-17T07:19:02.978 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:19:02.810226+0000 mgr.y (mgr.25096) 1 : cephadm [INF] inventory: adjusted host smithi179 addr 'smithi179' -> '172.21.15.179' 2024-09-17T07:19:02.978 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:02.814543+0000 mon.a (mon.0) 742 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:02.978 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:02.826464+0000 mon.c (mon.1) 58 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:19:02.978 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:02.854234+0000 mon.c (mon.1) 59 : audit [INF] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:19:02.979 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:02.854763+0000 mon.a (mon.0) 743 : audit [INF] from='mgr.25096 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:19:02.979 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:02.866929+0000 mon.c (mon.1) 60 : audit [INF] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:19:02.979 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:02.867438+0000 mon.a (mon.0) 744 : audit [INF] from='mgr.25096 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:19:03.255 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:03 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:19:03.042+0000 7fa47c45c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T07:19:03.255 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:03 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:19:03.115+0000 7fa47c45c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T07:19:03.295 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:01.081530+0000 mgr.x (mgr.34814) 86 : cluster [DBG] pgmap v44: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-17T07:19:03.295 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:19:01.646753+0000 mgr.x (mgr.34814) 87 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:19:03.295 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:01.647336+0000 mgr.x (mgr.34814) 88 : cluster [DBG] pgmap v45: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2024-09-17T07:19:03.295 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:01.647546+0000 mgr.x (mgr.34814) 89 : cluster [DBG] pgmap v46: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 119 B/s rd, 0 op/s 2024-09-17T07:19:03.295 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:19:01.770982+0000 mgr.x (mgr.34814) 90 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T07:19:03.295 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:19:01.771460+0000 mgr.x (mgr.34814) 91 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T07:19:03.295 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:19:01.773419+0000 mgr.x (mgr.34814) 92 : cephadm [INF] Failing over to other MGR 2024-09-17T07:19:03.295 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:01.968265+0000 mon.a (mon.0) 737 : cluster [WRN] Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T07:19:03.295 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:02.034371+0000 mon.a (mon.0) 738 : cluster [DBG] Standby manager daemon x started 2024-09-17T07:19:03.296 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:02.658601+0000 mon.a (mon.0) 739 : audit [INF] from='mgr.34814 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-17T07:19:03.296 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:02.658664+0000 mon.a (mon.0) 740 : cluster [DBG] mgrmap e20: y(active, starting, since 0.884112s), standbys: x 2024-09-17T07:19:03.296 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:02.661369+0000 mon.c (mon.1) 42 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:19:03.296 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:02.661548+0000 mon.c (mon.1) 43 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:19:03.296 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:02.661714+0000 mon.c (mon.1) 44 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:19:03.296 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:02.663112+0000 mon.c (mon.1) 45 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T07:19:03.296 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:02.663320+0000 mon.c (mon.1) 46 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T07:19:03.297 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:02.663589+0000 mon.c (mon.1) 47 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T07:19:03.297 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:02.663911+0000 mon.c (mon.1) 48 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T07:19:03.297 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:02.664252+0000 mon.c (mon.1) 49 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:19:03.297 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:02.664500+0000 mon.c (mon.1) 50 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:19:03.297 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:02.664699+0000 mon.c (mon.1) 51 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:19:03.297 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:02.665015+0000 mon.c (mon.1) 52 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:19:03.297 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:02.665385+0000 mon.c (mon.1) 53 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:19:03.297 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:02.665671+0000 mon.c (mon.1) 54 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:19:03.297 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:02.665978+0000 mon.c (mon.1) 55 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T07:19:03.298 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:02.666155+0000 mon.c (mon.1) 56 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T07:19:03.298 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:02.666579+0000 mon.c (mon.1) 57 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T07:19:03.298 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:02.798834+0000 mon.a (mon.0) 741 : cluster [INF] Manager daemon y is now available 2024-09-17T07:19:03.298 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:19:02.810226+0000 mgr.y (mgr.25096) 1 : cephadm [INF] inventory: adjusted host smithi179 addr 'smithi179' -> '172.21.15.179' 2024-09-17T07:19:03.298 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:02.814543+0000 mon.a (mon.0) 742 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:03.298 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:02.826464+0000 mon.c (mon.1) 58 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:19:03.298 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:02.854234+0000 mon.c (mon.1) 59 : audit [INF] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:19:03.298 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:02.854763+0000 mon.a (mon.0) 743 : audit [INF] from='mgr.25096 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:19:03.298 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:02.866929+0000 mon.c (mon.1) 60 : audit [INF] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:19:03.298 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:02.867438+0000 mon.a (mon.0) 744 : audit [INF] from='mgr.25096 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:19:03.300 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:01.081530+0000 mgr.x (mgr.34814) 86 : cluster [DBG] pgmap v44: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-17T07:19:03.301 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:19:01.646753+0000 mgr.x (mgr.34814) 87 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:19:03.301 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:01.647336+0000 mgr.x (mgr.34814) 88 : cluster [DBG] pgmap v45: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 96 B/s rd, 0 op/s 2024-09-17T07:19:03.301 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:01.647546+0000 mgr.x (mgr.34814) 89 : cluster [DBG] pgmap v46: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 119 B/s rd, 0 op/s 2024-09-17T07:19:03.301 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:19:01.770982+0000 mgr.x (mgr.34814) 90 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T07:19:03.301 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:19:01.771460+0000 mgr.x (mgr.34814) 91 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T07:19:03.301 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:19:01.773419+0000 mgr.x (mgr.34814) 92 : cephadm [INF] Failing over to other MGR 2024-09-17T07:19:03.301 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:01.968265+0000 mon.a (mon.0) 737 : cluster [WRN] Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T07:19:03.301 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:02.034371+0000 mon.a (mon.0) 738 : cluster [DBG] Standby manager daemon x started 2024-09-17T07:19:03.301 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:02.658601+0000 mon.a (mon.0) 739 : audit [INF] from='mgr.34814 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-17T07:19:03.302 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:02.658664+0000 mon.a (mon.0) 740 : cluster [DBG] mgrmap e20: y(active, starting, since 0.884112s), standbys: x 2024-09-17T07:19:03.302 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:02.661369+0000 mon.c (mon.1) 42 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:19:03.302 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:02.661548+0000 mon.c (mon.1) 43 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:19:03.302 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:02.661714+0000 mon.c (mon.1) 44 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:19:03.302 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:02.663112+0000 mon.c (mon.1) 45 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T07:19:03.302 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:02.663320+0000 mon.c (mon.1) 46 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T07:19:03.302 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:02.663589+0000 mon.c (mon.1) 47 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T07:19:03.302 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:02.663911+0000 mon.c (mon.1) 48 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T07:19:03.302 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:02.664252+0000 mon.c (mon.1) 49 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:19:03.302 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:02.664500+0000 mon.c (mon.1) 50 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:19:03.302 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:02.664699+0000 mon.c (mon.1) 51 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:19:03.302 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:02.665015+0000 mon.c (mon.1) 52 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:19:03.302 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:02.665385+0000 mon.c (mon.1) 53 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:19:03.302 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:02.665671+0000 mon.c (mon.1) 54 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:19:03.303 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:02.665978+0000 mon.c (mon.1) 55 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T07:19:03.303 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:02.666155+0000 mon.c (mon.1) 56 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T07:19:03.303 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:02.666579+0000 mon.c (mon.1) 57 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T07:19:03.303 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:02.798834+0000 mon.a (mon.0) 741 : cluster [INF] Manager daemon y is now available 2024-09-17T07:19:03.303 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:19:02.810226+0000 mgr.y (mgr.25096) 1 : cephadm [INF] inventory: adjusted host smithi179 addr 'smithi179' -> '172.21.15.179' 2024-09-17T07:19:03.303 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:02.814543+0000 mon.a (mon.0) 742 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:03.303 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:02.826464+0000 mon.c (mon.1) 58 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:19:03.303 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:02.854234+0000 mon.c (mon.1) 59 : audit [INF] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:19:03.303 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:02.854763+0000 mon.a (mon.0) 743 : audit [INF] from='mgr.25096 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:19:03.303 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:02.866929+0000 mon.c (mon.1) 60 : audit [INF] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:19:03.303 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:02.867438+0000 mon.a (mon.0) 744 : audit [INF] from='mgr.25096 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:19:03.613 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:03 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:19:03.277+0000 7fa47c45c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T07:19:03.614 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:03 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:19:03.430+0000 7fa47c45c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T07:19:04.064 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:03 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:19:03.770+0000 7fa47c45c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T07:19:04.064 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:03 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:19:03.845+0000 7fa47c45c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T07:19:04.065 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:04 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:19:04.062+0000 7fa47c45c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T07:19:04.612 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:04 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:03.552186+0000 mon.a (mon.0) 745 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:04.612 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:04 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:03.558788+0000 mon.a (mon.0) 746 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:04.612 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:04 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:03.662469+0000 mon.a (mon.0) 747 : cluster [DBG] mgrmap e21: y(active, since 1.88791s), standbys: x 2024-09-17T07:19:04.612 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:04 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:03.673590+0000 mgr.y (mgr.25096) 2 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:19:04.613 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:04 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:03.552186+0000 mon.a (mon.0) 745 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:04.613 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:04 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:03.558788+0000 mon.a (mon.0) 746 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:04.613 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:04 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:03.662469+0000 mon.a (mon.0) 747 : cluster [DBG] mgrmap e21: y(active, since 1.88791s), standbys: x 2024-09-17T07:19:04.613 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:04 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:03.673590+0000 mgr.y (mgr.25096) 2 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:19:04.864 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:04 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Scheduled restart job, restart counter is at 3. 2024-09-17T07:19:04.864 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:04 smithi179 systemd[1]: Stopped Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:19:04.864 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:04 smithi179 systemd[1]: Starting Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:19:04.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:04 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:03.552186+0000 mon.a (mon.0) 745 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:04.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:04 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:03.558788+0000 mon.a (mon.0) 746 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:04.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:04 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:03.662469+0000 mon.a (mon.0) 747 : cluster [DBG] mgrmap e21: y(active, since 1.88791s), standbys: x 2024-09-17T07:19:04.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:04 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:03.673590+0000 mgr.y (mgr.25096) 2 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:19:05.363 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:04 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:19:04.970+0000 7fa47c45c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T07:19:05.364 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:05 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:19:05.035+0000 7fa47c45c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T07:19:05.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:04.655297+0000 mon.a (mon.0) 748 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2024-09-17T07:19:05.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:04.655323+0000 mon.a (mon.0) 749 : cluster [INF] Cluster is now healthy 2024-09-17T07:19:05.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:04.664943+0000 mgr.y (mgr.25096) 3 : cluster [DBG] pgmap v4: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:19:05.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:04.655297+0000 mon.a (mon.0) 748 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2024-09-17T07:19:05.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:04.655323+0000 mon.a (mon.0) 749 : cluster [INF] Cluster is now healthy 2024-09-17T07:19:05.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:04.664943+0000 mgr.y (mgr.25096) 3 : cluster [DBG] pgmap v4: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:19:05.864 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:05 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:19:05.756+0000 7fa47c45c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T07:19:05.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:05 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:04.655297+0000 mon.a (mon.0) 748 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2024-09-17T07:19:05.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:05 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:04.655323+0000 mon.a (mon.0) 749 : cluster [INF] Cluster is now healthy 2024-09-17T07:19:05.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:05 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:04.664943+0000 mgr.y (mgr.25096) 3 : cluster [DBG] pgmap v4: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:19:06.220 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:05 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:19:05.872+0000 7fa47c45c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T07:19:06.220 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:19:06.012+0000 7fa47c45c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T07:19:06.220 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:19:06.073+0000 7fa47c45c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T07:19:06.220 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:19:06.219+0000 7fa47c45c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T07:19:06.513 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:19:06.287+0000 7fa47c45c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T07:19:06.513 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:19:06.346+0000 7fa47c45c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T07:19:06.514 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:19:06.512+0000 7fa47c45c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T07:19:06.556 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:19:06.351+0000 7fb716d55640 -1 log_channel(cephadm) log [ERR] : [17/Sep/2024:07:19:06] ENGINE Error in HTTPServer.serve 2024-09-17T07:19:06.556 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: Traceback (most recent call last): 2024-09-17T07:19:06.556 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T07:19:06.556 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: self._connections.run(self.expiration_interval) 2024-09-17T07:19:06.556 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T07:19:06.556 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: self._run(expiration_interval) 2024-09-17T07:19:06.556 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T07:19:06.556 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T07:19:06.556 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T07:19:06.557 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T07:19:06.557 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T07:19:06.557 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: s = self.context.wrap_socket( 2024-09-17T07:19:06.557 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T07:19:06.557 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: return self.sslsocket_class._create( 2024-09-17T07:19:06.557 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T07:19:06.557 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: self.do_handshake() 2024-09-17T07:19:06.557 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T07:19:06.557 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: self._sslobj.do_handshake() 2024-09-17T07:19:06.557 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T07:19:06.557 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:19:06.782 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:05.711031+0000 mon.a (mon.0) 750 : cluster [DBG] mgrmap e22: y(active, since 3s), standbys: x 2024-09-17T07:19:06.782 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:19:06.579+0000 7fa47c45c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T07:19:06.783 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:19:06.644+0000 7fa47c45c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T07:19:06.783 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:19:06.728+0000 7fa47c45c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T07:19:06.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:05.711031+0000 mon.a (mon.0) 750 : cluster [DBG] mgrmap e22: y(active, since 3s), standbys: x 2024-09-17T07:19:06.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:05.711031+0000 mon.a (mon.0) 750 : cluster [DBG] mgrmap e22: y(active, since 3s), standbys: x 2024-09-17T07:19:07.114 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:19:06.787+0000 7fa47c45c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T07:19:07.114 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:19:06.845+0000 7fa47c45c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T07:19:07.713 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:19:06.229587+0000 mgr.y (mgr.25096) 4 : cephadm [INF] [17/Sep/2024:07:19:06] ENGINE Bus STARTING 2024-09-17T07:19:07.713 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x[40948]: debug 2024-09-17T07:19:07.627+0000 7fa47c45c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T07:19:08.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:19:06.229587+0000 mgr.y (mgr.25096) 4 : cephadm [INF] [17/Sep/2024:07:19:06] ENGINE Bus STARTING 2024-09-17T07:19:08.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:19:06.349317+0000 mgr.y (mgr.25096) 5 : cephadm [INF] [17/Sep/2024:07:19:06] ENGINE Serving on https://172.21.15.44:7150 2024-09-17T07:19:08.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:19:06.352032+0000 mgr.y (mgr.25096) 6 : cephadm [ERR] [17/Sep/2024:07:19:06] ENGINE Error in HTTPServer.serve 2024-09-17T07:19:08.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: Traceback (most recent call last): 2024-09-17T07:19:08.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T07:19:08.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: self._connections.run(self.expiration_interval) 2024-09-17T07:19:08.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T07:19:08.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: self._run(expiration_interval) 2024-09-17T07:19:08.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T07:19:08.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T07:19:08.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T07:19:08.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T07:19:08.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T07:19:08.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: s = self.context.wrap_socket( 2024-09-17T07:19:08.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T07:19:08.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: return self.sslsocket_class._create( 2024-09-17T07:19:08.077 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T07:19:08.077 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: self.do_handshake() 2024-09-17T07:19:08.077 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T07:19:08.077 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: self._sslobj.do_handshake() 2024-09-17T07:19:08.077 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T07:19:08.077 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: 2024-09-17T07:19:08.077 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:19:06.450537+0000 mgr.y (mgr.25096) 7 : cephadm [INF] [17/Sep/2024:07:19:06] ENGINE Serving on http://172.21.15.44:8765 2024-09-17T07:19:08.077 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:19:06.450640+0000 mgr.y (mgr.25096) 8 : cephadm [INF] [17/Sep/2024:07:19:06] ENGINE Bus STARTED 2024-09-17T07:19:08.077 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:06.665514+0000 mgr.y (mgr.25096) 9 : cluster [DBG] pgmap v5: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:19:08.077 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:07.630624+0000 mon.a (mon.0) 751 : cluster [DBG] Standby manager daemon x restarted 2024-09-17T07:19:08.078 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:19:06.229587+0000 mgr.y (mgr.25096) 4 : cephadm [INF] [17/Sep/2024:07:19:06] ENGINE Bus STARTING 2024-09-17T07:19:08.078 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:19:06.349317+0000 mgr.y (mgr.25096) 5 : cephadm [INF] [17/Sep/2024:07:19:06] ENGINE Serving on https://172.21.15.44:7150 2024-09-17T07:19:08.078 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:19:06.352032+0000 mgr.y (mgr.25096) 6 : cephadm [ERR] [17/Sep/2024:07:19:06] ENGINE Error in HTTPServer.serve 2024-09-17T07:19:08.078 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: Traceback (most recent call last): 2024-09-17T07:19:08.078 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T07:19:08.078 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: self._connections.run(self.expiration_interval) 2024-09-17T07:19:08.078 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T07:19:08.078 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: self._run(expiration_interval) 2024-09-17T07:19:08.078 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T07:19:08.078 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T07:19:08.078 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T07:19:08.078 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T07:19:08.078 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T07:19:08.078 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: s = self.context.wrap_socket( 2024-09-17T07:19:08.079 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T07:19:08.079 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: return self.sslsocket_class._create( 2024-09-17T07:19:08.079 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T07:19:08.079 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: self.do_handshake() 2024-09-17T07:19:08.079 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T07:19:08.079 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: self._sslobj.do_handshake() 2024-09-17T07:19:08.079 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T07:19:08.079 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: 2024-09-17T07:19:08.079 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:19:06.450537+0000 mgr.y (mgr.25096) 7 : cephadm [INF] [17/Sep/2024:07:19:06] ENGINE Serving on http://172.21.15.44:8765 2024-09-17T07:19:08.079 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:19:06.450640+0000 mgr.y (mgr.25096) 8 : cephadm [INF] [17/Sep/2024:07:19:06] ENGINE Bus STARTED 2024-09-17T07:19:08.079 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:06.665514+0000 mgr.y (mgr.25096) 9 : cluster [DBG] pgmap v5: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:19:08.079 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:07.630624+0000 mon.a (mon.0) 751 : cluster [DBG] Standby manager daemon x restarted 2024-09-17T07:19:08.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:07.630774+0000 mon.a (mon.0) 752 : cluster [DBG] Standby manager daemon x started 2024-09-17T07:19:08.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:07.630935+0000 mon.b (mon.2) 80 : audit [DBG] from='mgr.? 172.21.15.179:0/1688854771' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T07:19:08.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:07.632601+0000 mon.b (mon.2) 81 : audit [DBG] from='mgr.? 172.21.15.179:0/1688854771' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T07:19:08.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:07.633296+0000 mon.b (mon.2) 82 : audit [DBG] from='mgr.? 172.21.15.179:0/1688854771' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T07:19:08.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:07.633554+0000 mon.b (mon.2) 83 : audit [DBG] from='mgr.? 172.21.15.179:0/1688854771' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T07:19:08.080 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:07.630774+0000 mon.a (mon.0) 752 : cluster [DBG] Standby manager daemon x started 2024-09-17T07:19:08.080 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:07.630935+0000 mon.b (mon.2) 80 : audit [DBG] from='mgr.? 172.21.15.179:0/1688854771' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T07:19:08.080 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:07.632601+0000 mon.b (mon.2) 81 : audit [DBG] from='mgr.? 172.21.15.179:0/1688854771' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T07:19:08.080 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:07.633296+0000 mon.b (mon.2) 82 : audit [DBG] from='mgr.? 172.21.15.179:0/1688854771' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T07:19:08.081 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:07 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:07.633554+0000 mon.b (mon.2) 83 : audit [DBG] from='mgr.? 172.21.15.179:0/1688854771' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T07:19:08.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:19:06.349317+0000 mgr.y (mgr.25096) 5 : cephadm [INF] [17/Sep/2024:07:19:06] ENGINE Serving on https://172.21.15.44:7150 2024-09-17T07:19:08.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:19:06.352032+0000 mgr.y (mgr.25096) 6 : cephadm [ERR] [17/Sep/2024:07:19:06] ENGINE Error in HTTPServer.serve 2024-09-17T07:19:08.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: Traceback (most recent call last): 2024-09-17T07:19:08.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T07:19:08.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: self._connections.run(self.expiration_interval) 2024-09-17T07:19:08.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T07:19:08.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: self._run(expiration_interval) 2024-09-17T07:19:08.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T07:19:08.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T07:19:08.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T07:19:08.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T07:19:08.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T07:19:08.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: s = self.context.wrap_socket( 2024-09-17T07:19:08.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T07:19:08.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: return self.sslsocket_class._create( 2024-09-17T07:19:08.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T07:19:08.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: self.do_handshake() 2024-09-17T07:19:08.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T07:19:08.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: self._sslobj.do_handshake() 2024-09-17T07:19:08.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T07:19:08.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: 2024-09-17T07:19:08.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:19:06.450537+0000 mgr.y (mgr.25096) 7 : cephadm [INF] [17/Sep/2024:07:19:06] ENGINE Serving on http://172.21.15.44:8765 2024-09-17T07:19:08.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:19:06.450640+0000 mgr.y (mgr.25096) 8 : cephadm [INF] [17/Sep/2024:07:19:06] ENGINE Bus STARTED 2024-09-17T07:19:08.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:06.665514+0000 mgr.y (mgr.25096) 9 : cluster [DBG] pgmap v5: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:19:08.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:07.630624+0000 mon.a (mon.0) 751 : cluster [DBG] Standby manager daemon x restarted 2024-09-17T07:19:08.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:07.630774+0000 mon.a (mon.0) 752 : cluster [DBG] Standby manager daemon x started 2024-09-17T07:19:08.117 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:07.630935+0000 mon.b (mon.2) 80 : audit [DBG] from='mgr.? 172.21.15.179:0/1688854771' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T07:19:08.117 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:07.632601+0000 mon.b (mon.2) 81 : audit [DBG] from='mgr.? 172.21.15.179:0/1688854771' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T07:19:08.117 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:07.633296+0000 mon.b (mon.2) 82 : audit [DBG] from='mgr.? 172.21.15.179:0/1688854771' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T07:19:08.117 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:07.633554+0000 mon.b (mon.2) 83 : audit [DBG] from='mgr.? 172.21.15.179:0/1688854771' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T07:19:09.073 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:07.723932+0000 mon.a (mon.0) 753 : cluster [DBG] mgrmap e23: y(active, since 5s), standbys: x 2024-09-17T07:19:09.073 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:08.175701+0000 mon.a (mon.0) 754 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:09.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:08.182661+0000 mon.a (mon.0) 755 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:09.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:08.701731+0000 mon.a (mon.0) 756 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:09.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:08.707240+0000 mon.a (mon.0) 757 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:09.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:07.723932+0000 mon.a (mon.0) 753 : cluster [DBG] mgrmap e23: y(active, since 5s), standbys: x 2024-09-17T07:19:09.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:08.175701+0000 mon.a (mon.0) 754 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:09.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:08.182661+0000 mon.a (mon.0) 755 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:09.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:08.701731+0000 mon.a (mon.0) 756 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:09.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:08.707240+0000 mon.a (mon.0) 757 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:09.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:07.723932+0000 mon.a (mon.0) 753 : cluster [DBG] mgrmap e23: y(active, since 5s), standbys: x 2024-09-17T07:19:09.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:08.175701+0000 mon.a (mon.0) 754 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:09.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:08.182661+0000 mon.a (mon.0) 755 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:09.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:08.701731+0000 mon.a (mon.0) 756 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:09.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:08.707240+0000 mon.a (mon.0) 757 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:09.477 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:09 smithi179 podman[45362]: 2024-09-17 07:19:09.228937312 +0000 UTC m=+0.020368014 image pull de242295e2257c37c8cadfd962369228f8f10b2d48a44259b65fef44ad4f6490 docker.io/prom/prometheus:v2.18.1 2024-09-17T07:19:09.477 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:09 smithi179 podman[45362]: 2024-09-17 07:19:09.349708671 +0000 UTC m=+0.141139341 container create a8a37fb9b11e780981015fb8bd03878d66d7b46ba3fb9d3e1ffcc21822e69d39 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:19:09.743 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:08.666003+0000 mgr.y (mgr.25096) 10 : cluster [DBG] pgmap v6: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:19:09.744 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:09 smithi179 podman[45362]: 2024-09-17 07:19:09.618354712 +0000 UTC m=+0.409785385 container init a8a37fb9b11e780981015fb8bd03878d66d7b46ba3fb9d3e1ffcc21822e69d39 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:19:09.744 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:09 smithi179 podman[45362]: 2024-09-17 07:19:09.621261562 +0000 UTC m=+0.412692235 container start a8a37fb9b11e780981015fb8bd03878d66d7b46ba3fb9d3e1ffcc21822e69d39 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:19:09.744 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[45377]: level=info ts=2024-09-17T07:19:09.650Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-17T07:19:09.744 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[45377]: level=info ts=2024-09-17T07:19:09.650Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2024-09-17T07:19:09.744 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[45377]: level=info ts=2024-09-17T07:19:09.650Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2024-09-17T07:19:09.744 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[45377]: level=info ts=2024-09-17T07:19:09.650Z caller=main.go:339 host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi179 (none))" 2024-09-17T07:19:09.744 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[45377]: level=info ts=2024-09-17T07:19:09.650Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T07:19:09.745 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[45377]: level=info ts=2024-09-17T07:19:09.650Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T07:19:09.745 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[45377]: level=info ts=2024-09-17T07:19:09.652Z caller=main.go:678 msg="Starting TSDB ..." 2024-09-17T07:19:09.745 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[45377]: level=info ts=2024-09-17T07:19:09.652Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2024-09-17T07:19:09.745 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[45377]: level=info ts=2024-09-17T07:19:09.655Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2024-09-17T07:19:09.745 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[45377]: level=info ts=2024-09-17T07:19:09.655Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=5 2024-09-17T07:19:09.745 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[45377]: level=info ts=2024-09-17T07:19:09.661Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=5 2024-09-17T07:19:09.745 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[45377]: level=info ts=2024-09-17T07:19:09.661Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=5 2024-09-17T07:19:09.745 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[45377]: level=info ts=2024-09-17T07:19:09.661Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=5 2024-09-17T07:19:09.745 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[45377]: level=info ts=2024-09-17T07:19:09.661Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=5 2024-09-17T07:19:09.745 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[45377]: level=info ts=2024-09-17T07:19:09.661Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=5 2024-09-17T07:19:09.745 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[45377]: level=info ts=2024-09-17T07:19:09.661Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=6.595562ms 2024-09-17T07:19:09.745 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[45377]: level=info ts=2024-09-17T07:19:09.663Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2024-09-17T07:19:09.746 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[45377]: level=info ts=2024-09-17T07:19:09.663Z caller=main.go:695 msg="TSDB started" 2024-09-17T07:19:09.746 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[45377]: level=info ts=2024-09-17T07:19:09.663Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T07:19:09.746 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[45377]: level=info ts=2024-09-17T07:19:09.663Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2024-09-17T07:19:09.746 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[45377]: level=info ts=2024-09-17T07:19:09.663Z caller=main.go:561 msg="Stopping notify discovery manager..." 2024-09-17T07:19:09.746 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[45377]: level=info ts=2024-09-17T07:19:09.663Z caller=main.go:583 msg="Stopping scrape manager..." 2024-09-17T07:19:09.746 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[45377]: level=info ts=2024-09-17T07:19:09.663Z caller=main.go:557 msg="Notify discovery manager stopped" 2024-09-17T07:19:09.746 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[45377]: level=info ts=2024-09-17T07:19:09.663Z caller=main.go:543 msg="Scrape discovery manager stopped" 2024-09-17T07:19:09.746 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[45377]: level=info ts=2024-09-17T07:19:09.664Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2024-09-17T07:19:09.746 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[45377]: level=info ts=2024-09-17T07:19:09.664Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2024-09-17T07:19:09.746 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[45377]: level=info ts=2024-09-17T07:19:09.664Z caller=main.go:577 msg="Scrape manager stopped" 2024-09-17T07:19:09.746 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[45377]: level=info ts=2024-09-17T07:19:09.664Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2024-09-17T07:19:09.746 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[45377]: level=info ts=2024-09-17T07:19:09.664Z caller=main.go:749 msg="Notifier manager stopped" 2024-09-17T07:19:09.746 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[45377]: level=error ts=2024-09-17T07:19:09.664Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 14: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain\n line 31: field http_sd_configs not found in type config.plain\n line 45: field http_sd_configs not found in type config.plain\n line 49: field http_sd_configs not found in type config.plain" 2024-09-17T07:19:09.746 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:09 smithi179 bash[45362]: a8a37fb9b11e780981015fb8bd03878d66d7b46ba3fb9d3e1ffcc21822e69d39 2024-09-17T07:19:10.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:08.666003+0000 mgr.y (mgr.25096) 10 : cluster [DBG] pgmap v6: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:19:10.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:09 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:08.666003+0000 mgr.y (mgr.25096) 10 : cluster [DBG] pgmap v6: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:19:10.098 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:09 smithi179 systemd[1]: Started Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:19:10.771 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:10 smithi179 podman[45394]: 2024-09-17 07:19:10.473072956 +0000 UTC m=+0.800585420 container remove a8a37fb9b11e780981015fb8bd03878d66d7b46ba3fb9d3e1ffcc21822e69d39 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:19:10.771 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:10 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2024-09-17T07:19:10.771 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:10 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Failed with result 'exit-code'. 2024-09-17T07:19:11.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:11 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:10.666743+0000 mgr.y (mgr.25096) 11 : cluster [DBG] pgmap v7: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-17T07:19:12.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:10.666743+0000 mgr.y (mgr.25096) 11 : cluster [DBG] pgmap v7: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-17T07:19:12.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:10.666743+0000 mgr.y (mgr.25096) 11 : cluster [DBG] pgmap v7: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-17T07:19:13.424 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:12.179489+0000 mon.a (mon.0) 758 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:13.425 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:12.186084+0000 mon.a (mon.0) 759 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:13.425 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:12.667102+0000 mgr.y (mgr.25096) 12 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-17T07:19:13.425 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:12.179489+0000 mon.a (mon.0) 758 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:13.425 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:12.186084+0000 mon.a (mon.0) 759 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:13.425 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:13 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:12.667102+0000 mgr.y (mgr.25096) 12 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-17T07:19:13.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:13 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:12.179489+0000 mon.a (mon.0) 758 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:13.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:13 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:12.186084+0000 mon.a (mon.0) 759 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:13.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:13 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:12.667102+0000 mgr.y (mgr.25096) 12 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-17T07:19:13.696 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-09-17T07:19:14.415 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T07:19:14.415 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.a smithi044 *:9093,9094 running (59s) 6s ago 5m 13.0M - 0.20.0 0881eb8f169f 0473d4b023ad 2024-09-17T07:19:14.415 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.a smithi179 *:3000 running (30s) 0s ago 5m 26.5M - 6.7.4 557c83e11646 43ab0e03dcac 2024-09-17T07:19:14.415 INFO:teuthology.orchestra.run.smithi044.stdout:iscsi.foo.smithi044.elmtfk smithi044 unknown 6s ago 4m - - 2024-09-17T07:19:14.416 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.x smithi179 *:8443 running (95s) 0s ago 9m 111M - 18.2.4-855-g19e2a2be 980a0bfcd684 c9b478bf9874 2024-09-17T07:19:14.416 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.y smithi044 *:8443,8765 running (22s) 6s ago 11m 457M - 18.2.4-855-g19e2a2be 980a0bfcd684 8080dd318a1c 2024-09-17T07:19:14.416 INFO:teuthology.orchestra.run.smithi044.stdout:mon.a smithi044 running (11m) 6s ago 11m 56.1M 2048M 16.2.0 24ecd6d5f14c 7905efa25a24 2024-09-17T07:19:14.416 INFO:teuthology.orchestra.run.smithi044.stdout:mon.b smithi179 running (9m) 0s ago 9m 53.1M 2048M 16.2.0 24ecd6d5f14c 198f81fe2641 2024-09-17T07:19:14.416 INFO:teuthology.orchestra.run.smithi044.stdout:mon.c smithi044 running (10m) 6s ago 10m 48.8M 2048M 16.2.0 24ecd6d5f14c e0d29cfe87a2 2024-09-17T07:19:14.416 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.a smithi044 *:9100 running (67s) 6s ago 6m 12.9M - 0.18.1 e5a616e4b9cf 8a2cde5bd9ed 2024-09-17T07:19:14.416 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.b smithi179 *:9100 running (42s) 0s ago 6m 4710k - 0.18.1 e5a616e4b9cf 04c63b5cb524 2024-09-17T07:19:14.416 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (9m) 6s ago 9m 47.9M 4096M 16.2.0 24ecd6d5f14c 4517de6029ac 2024-09-17T07:19:14.416 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (9m) 6s ago 9m 45.3M 4096M 16.2.0 24ecd6d5f14c 2d5122cbeff9 2024-09-17T07:19:14.416 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (8m) 6s ago 8m 42.9M 4096M 16.2.0 24ecd6d5f14c 7fd2f30be874 2024-09-17T07:19:14.416 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi044 running (8m) 6s ago 8m 48.1M 4096M 16.2.0 24ecd6d5f14c 2b7d623395a7 2024-09-17T07:19:14.416 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi179 running (8m) 0s ago 8m 51.1M 4096M 16.2.0 24ecd6d5f14c 911e6f47c822 2024-09-17T07:19:14.416 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi179 running (7m) 0s ago 7m 47.3M 4096M 16.2.0 24ecd6d5f14c e56060742dff 2024-09-17T07:19:14.416 INFO:teuthology.orchestra.run.smithi044.stdout:osd.6 smithi179 running (7m) 0s ago 7m 46.7M 4096M 16.2.0 24ecd6d5f14c 7a5c30b68cbc 2024-09-17T07:19:14.417 INFO:teuthology.orchestra.run.smithi044.stdout:osd.7 smithi179 running (7m) 0s ago 7m 52.8M 4096M 16.2.0 24ecd6d5f14c b0d186fbf8a6 2024-09-17T07:19:14.417 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.a smithi179 *:9095 unknown 0s ago 6m - - 2024-09-17T07:19:14.417 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi044.nhyhnh smithi044 *:8000 running (4m) 6s ago 4m 37.7M - 16.2.0 24ecd6d5f14c 9b455479614f 2024-09-17T07:19:14.417 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi179.tqlzyy smithi179 *:8000 running (4m) 0s ago 4m 40.5M - 16.2.0 24ecd6d5f14c c8d1693241ce 2024-09-17T07:19:14.417 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi044.vnufiz smithi044 *:80 running (4m) 6s ago 4m 37.3M - 16.2.0 24ecd6d5f14c 6eae2bff5ec8 2024-09-17T07:19:14.417 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi179.ecjhll smithi179 *:80 running (4m) 0s ago 4m 40.7M - 16.2.0 24ecd6d5f14c 253109ee0394 2024-09-17T07:19:14.417 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:13.683487+0000 mgr.y (mgr.25096) 13 : audit [DBG] from='client.15237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:19:14.417 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:13.900724+0000 mon.a (mon.0) 760 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:14.418 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:13.907219+0000 mon.a (mon.0) 761 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:14.418 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:13.683487+0000 mgr.y (mgr.25096) 13 : audit [DBG] from='client.15237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:19:14.418 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:13.900724+0000 mon.a (mon.0) 760 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:14.418 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:13.907219+0000 mon.a (mon.0) 761 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:14.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:14 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:13.683487+0000 mgr.y (mgr.25096) 13 : audit [DBG] from='client.15237 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:19:14.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:14 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:13.900724+0000 mon.a (mon.0) 760 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:14.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:14 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:13.907219+0000 mon.a (mon.0) 761 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:14.848 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:19:14.848 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-09-17T07:19:14.849 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 3 2024-09-17T07:19:14.849 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:19:14.849 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-09-17T07:19:14.849 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T07:19:14.849 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:19:14.849 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-09-17T07:19:14.849 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2024-09-17T07:19:14.849 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:19:14.849 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": {}, 2024-09-17T07:19:14.849 INFO:teuthology.orchestra.run.smithi044.stdout: "rgw": { 2024-09-17T07:19:14.849 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2024-09-17T07:19:14.849 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:19:14.850 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-09-17T07:19:14.850 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 15, 2024-09-17T07:19:14.850 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T07:19:14.850 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-09-17T07:19:14.850 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:19:15.220 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:19:15.221 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9", 2024-09-17T07:19:15.221 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-09-17T07:19:15.221 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T07:19:15.221 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [ 2024-09-17T07:19:15.221 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr" 2024-09-17T07:19:15.221 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-09-17T07:19:15.221 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "2/23 daemons upgraded", 2024-09-17T07:19:15.221 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "", 2024-09-17T07:19:15.221 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-09-17T07:19:15.221 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:19:15.621 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_WARN 2 failed cephadm daemon(s) 2024-09-17T07:19:15.621 INFO:teuthology.orchestra.run.smithi044.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T07:19:15.621 INFO:teuthology.orchestra.run.smithi044.stdout: daemon iscsi.foo.smithi044.elmtfk on smithi044 is in unknown state 2024-09-17T07:19:15.621 INFO:teuthology.orchestra.run.smithi044.stdout: daemon prometheus.a on smithi179 is in unknown state 2024-09-17T07:19:15.702 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:14.044270+0000 mgr.y (mgr.25096) 14 : audit [DBG] from='client.15243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:19:15.703 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:14.405829+0000 mgr.y (mgr.25096) 15 : audit [DBG] from='client.25168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:19:15.703 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:14.422759+0000 mon.a (mon.0) 762 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:15.703 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:14.430992+0000 mon.a (mon.0) 763 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:15.703 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:14.433046+0000 mon.c (mon.1) 61 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:19:15.703 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:14.433892+0000 mon.c (mon.1) 62 : audit [INF] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:19:15.703 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:19:14.434624+0000 mgr.y (mgr.25096) 16 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:19:15.703 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:14.435695+0000 mgr.y (mgr.25096) 17 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T07:19:15.703 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:14.441370+0000 mon.a (mon.0) 764 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:15.703 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:19:14.519056+0000 mgr.y (mgr.25096) 18 : cephadm [INF] Reconfiguring iscsi.foo.smithi044.elmtfk (dependencies changed)... 2024-09-17T07:19:15.703 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:14.519463+0000 mon.c (mon.1) 63 : audit [INF] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi044.elmtfk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T07:19:15.703 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:14.519892+0000 mon.a (mon.0) 765 : audit [INF] from='mgr.25096 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi044.elmtfk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T07:19:15.704 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:14.525283+0000 mon.c (mon.1) 64 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:19:15.704 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:19:14.526167+0000 mgr.y (mgr.25096) 19 : cephadm [INF] Reconfiguring daemon iscsi.foo.smithi044.elmtfk on smithi044 2024-09-17T07:19:15.704 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:14.044270+0000 mgr.y (mgr.25096) 14 : audit [DBG] from='client.15243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:19:15.704 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:14.405829+0000 mgr.y (mgr.25096) 15 : audit [DBG] from='client.25168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:19:15.704 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:14.422759+0000 mon.a (mon.0) 762 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:15.705 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:14.430992+0000 mon.a (mon.0) 763 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:15.705 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:14.433046+0000 mon.c (mon.1) 61 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:19:15.705 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:14.433892+0000 mon.c (mon.1) 62 : audit [INF] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:19:15.705 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:19:14.434624+0000 mgr.y (mgr.25096) 16 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:19:15.705 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:14.435695+0000 mgr.y (mgr.25096) 17 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T07:19:15.705 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:14.441370+0000 mon.a (mon.0) 764 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:15.705 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:19:14.519056+0000 mgr.y (mgr.25096) 18 : cephadm [INF] Reconfiguring iscsi.foo.smithi044.elmtfk (dependencies changed)... 2024-09-17T07:19:15.705 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:14.519463+0000 mon.c (mon.1) 63 : audit [INF] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi044.elmtfk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T07:19:15.705 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:14.519892+0000 mon.a (mon.0) 765 : audit [INF] from='mgr.25096 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi044.elmtfk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T07:19:15.705 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:14.525283+0000 mon.c (mon.1) 64 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:19:15.705 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:19:14.526167+0000 mgr.y (mgr.25096) 19 : cephadm [INF] Reconfiguring daemon iscsi.foo.smithi044.elmtfk on smithi044 2024-09-17T07:19:15.706 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:14.737296+0000 mon.c (mon.1) 65 : audit [DBG] from='client.? 172.21.15.44:0/1703861598' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:19:15.706 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:14.848103+0000 mon.a (mon.0) 766 : audit [DBG] from='client.? 172.21.15.44:0/3319859277' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:19:15.706 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:15.087357+0000 mon.c (mon.1) 66 : audit [INF] from='client.? 172.21.15.44:0/484653630' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:6800/2448217567"}]: access denied 2024-09-17T07:19:15.706 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:14.737296+0000 mon.c (mon.1) 65 : audit [DBG] from='client.? 172.21.15.44:0/1703861598' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:19:15.706 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:14.848103+0000 mon.a (mon.0) 766 : audit [DBG] from='client.? 172.21.15.44:0/3319859277' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:19:15.706 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:15.087357+0000 mon.c (mon.1) 66 : audit [INF] from='client.? 172.21.15.44:0/484653630' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:6800/2448217567"}]: access denied 2024-09-17T07:19:15.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:14.044270+0000 mgr.y (mgr.25096) 14 : audit [DBG] from='client.15243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:19:15.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:14.405829+0000 mgr.y (mgr.25096) 15 : audit [DBG] from='client.25168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:19:15.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:14.422759+0000 mon.a (mon.0) 762 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:15.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:14.430992+0000 mon.a (mon.0) 763 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:15.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:14.433046+0000 mon.c (mon.1) 61 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:19:15.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:14.433892+0000 mon.c (mon.1) 62 : audit [INF] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:19:15.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:19:14.434624+0000 mgr.y (mgr.25096) 16 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:19:15.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:14.435695+0000 mgr.y (mgr.25096) 17 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T07:19:15.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:14.441370+0000 mon.a (mon.0) 764 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:15.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:19:14.519056+0000 mgr.y (mgr.25096) 18 : cephadm [INF] Reconfiguring iscsi.foo.smithi044.elmtfk (dependencies changed)... 2024-09-17T07:19:15.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:14.519463+0000 mon.c (mon.1) 63 : audit [INF] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi044.elmtfk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T07:19:15.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:14.519892+0000 mon.a (mon.0) 765 : audit [INF] from='mgr.25096 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi044.elmtfk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T07:19:15.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:14.525283+0000 mon.c (mon.1) 64 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:19:15.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:19:14.526167+0000 mgr.y (mgr.25096) 19 : cephadm [INF] Reconfiguring daemon iscsi.foo.smithi044.elmtfk on smithi044 2024-09-17T07:19:15.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:14.737296+0000 mon.c (mon.1) 65 : audit [DBG] from='client.? 172.21.15.44:0/1703861598' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:19:15.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:14.848103+0000 mon.a (mon.0) 766 : audit [DBG] from='client.? 172.21.15.44:0/3319859277' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:19:15.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:15.087357+0000 mon.c (mon.1) 66 : audit [INF] from='client.? 172.21.15.44:0/484653630' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:6800/2448217567"}]: access denied 2024-09-17T07:19:16.664 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:15.219943+0000 mgr.y (mgr.25096) 20 : audit [DBG] from='client.15285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:19:16.665 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:15.428776+0000 mon.a (mon.0) 767 : cluster [WRN] Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T07:19:16.665 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:15.621684+0000 mon.a (mon.0) 768 : audit [DBG] from='client.? 172.21.15.44:0/2947518699' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:19:16.665 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:15.219943+0000 mgr.y (mgr.25096) 20 : audit [DBG] from='client.15285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:19:16.665 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:15.428776+0000 mon.a (mon.0) 767 : cluster [WRN] Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T07:19:16.665 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:15.621684+0000 mon.a (mon.0) 768 : audit [DBG] from='client.? 172.21.15.44:0/2947518699' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:19:16.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:16 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:15.219943+0000 mgr.y (mgr.25096) 20 : audit [DBG] from='client.15285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:19:16.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:16 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:15.428776+0000 mon.a (mon.0) 767 : cluster [WRN] Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T07:19:16.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:16 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:15.621684+0000 mon.a (mon.0) 768 : audit [DBG] from='client.? 172.21.15.44:0/2947518699' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:19:17.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:17 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:16.436052+0000 mgr.y (mgr.25096) 21 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-17T07:19:17.574 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:17 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:16.436052+0000 mgr.y (mgr.25096) 21 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-17T07:19:17.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:16.436052+0000 mgr.y (mgr.25096) 21 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-17T07:19:19.939 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:18.436446+0000 mgr.y (mgr.25096) 22 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-17T07:19:19.939 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:18.753423+0000 mon.a (mon.0) 769 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:19.940 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:18.759102+0000 mon.a (mon.0) 770 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:19.940 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:19:18.762771+0000 mgr.y (mgr.25096) 23 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T07:19:19.940 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:19.251336+0000 mon.c (mon.1) 67 : audit [DBG] from='client.? 172.21.15.44:0/1267127712' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:19:19.940 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:19.585835+0000 mon.c (mon.1) 68 : audit [INF] from='client.? 172.21.15.44:0/2406612472' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:6800/2448217567"}]: access denied 2024-09-17T07:19:19.940 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:18.436446+0000 mgr.y (mgr.25096) 22 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-17T07:19:19.940 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:18.753423+0000 mon.a (mon.0) 769 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:19.940 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:18.759102+0000 mon.a (mon.0) 770 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:19.940 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:19:18.762771+0000 mgr.y (mgr.25096) 23 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T07:19:19.940 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:19.251336+0000 mon.c (mon.1) 67 : audit [DBG] from='client.? 172.21.15.44:0/1267127712' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:19:19.940 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:19.585835+0000 mon.c (mon.1) 68 : audit [INF] from='client.? 172.21.15.44:0/2406612472' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:6800/2448217567"}]: access denied 2024-09-17T07:19:20.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:19 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:18.436446+0000 mgr.y (mgr.25096) 22 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-17T07:19:20.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:19 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:18.753423+0000 mon.a (mon.0) 769 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:20.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:19 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:18.759102+0000 mon.a (mon.0) 770 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:20.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:19 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:19:18.762771+0000 mgr.y (mgr.25096) 23 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T07:19:20.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:19 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:19.251336+0000 mon.c (mon.1) 67 : audit [DBG] from='client.? 172.21.15.44:0/1267127712' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:19:20.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:19 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:19.585835+0000 mon.c (mon.1) 68 : audit [INF] from='client.? 172.21.15.44:0/2406612472' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:6800/2448217567"}]: access denied 2024-09-17T07:19:20.765 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:20 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Scheduled restart job, restart counter is at 4. 2024-09-17T07:19:20.765 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:20 smithi179 systemd[1]: Stopped Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:19:20.765 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:20 smithi179 systemd[1]: Starting Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:19:21.014 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:19:19.020490+0000 mgr.y (mgr.25096) 24 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi179 2024-09-17T07:19:21.014 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:19:19.020490+0000 mgr.y (mgr.25096) 24 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi179 2024-09-17T07:19:21.065 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:19:19.020490+0000 mgr.y (mgr.25096) 24 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi179 2024-09-17T07:19:21.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:21 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:20.437175+0000 mgr.y (mgr.25096) 25 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T07:19:21.864 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:21 smithi179 podman[46284]: 2024-09-17 07:19:21.552110132 +0000 UTC m=+0.023916111 image pull de242295e2257c37c8cadfd962369228f8f10b2d48a44259b65fef44ad4f6490 docker.io/prom/prometheus:v2.18.1 2024-09-17T07:19:22.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:21 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:20.437175+0000 mgr.y (mgr.25096) 25 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T07:19:22.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:21 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:20.437175+0000 mgr.y (mgr.25096) 25 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T07:19:22.305 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-17T07:19:22.305 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Unit process 46228 (bash) remains running after unit stopped. 2024-09-17T07:19:22.305 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Unit process 46284 (podman) remains running after unit stopped. 2024-09-17T07:19:22.305 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 systemd[1]: Stopped Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:19:22.305 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 systemd[1]: Starting Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:19:22.305 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 podman[46284]: 2024-09-17 07:19:22.208917138 +0000 UTC m=+0.680723104 container create 7068e45188892c0a73ea313f590370a6306b1e31caff10942abc9dc282caa606 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:19:22.614 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 podman[46284]: 2024-09-17 07:19:22.411120635 +0000 UTC m=+0.882926611 container init 7068e45188892c0a73ea313f590370a6306b1e31caff10942abc9dc282caa606 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:19:22.614 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 podman[46284]: 2024-09-17 07:19:22.414009077 +0000 UTC m=+0.885815044 container start 7068e45188892c0a73ea313f590370a6306b1e31caff10942abc9dc282caa606 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:19:22.614 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46328]: level=info ts=2024-09-17T07:19:22.444Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-17T07:19:22.614 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46328]: level=info ts=2024-09-17T07:19:22.444Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2024-09-17T07:19:22.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46328]: level=info ts=2024-09-17T07:19:22.444Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2024-09-17T07:19:22.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46328]: level=info ts=2024-09-17T07:19:22.444Z caller=main.go:339 host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi179 (none))" 2024-09-17T07:19:22.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46328]: level=info ts=2024-09-17T07:19:22.444Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T07:19:22.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46328]: level=info ts=2024-09-17T07:19:22.444Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T07:19:22.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46328]: level=info ts=2024-09-17T07:19:22.446Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2024-09-17T07:19:22.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46328]: level=info ts=2024-09-17T07:19:22.446Z caller=main.go:678 msg="Starting TSDB ..." 2024-09-17T07:19:22.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46328]: level=info ts=2024-09-17T07:19:22.449Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2024-09-17T07:19:22.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46328]: level=info ts=2024-09-17T07:19:22.449Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=6 2024-09-17T07:19:22.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46328]: level=info ts=2024-09-17T07:19:22.455Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=6 2024-09-17T07:19:22.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46328]: level=info ts=2024-09-17T07:19:22.456Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=6 2024-09-17T07:19:22.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46328]: level=info ts=2024-09-17T07:19:22.456Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=6 2024-09-17T07:19:22.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46328]: level=info ts=2024-09-17T07:19:22.456Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=6 2024-09-17T07:19:22.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46328]: level=info ts=2024-09-17T07:19:22.458Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=6 2024-09-17T07:19:22.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46328]: level=info ts=2024-09-17T07:19:22.458Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=6 2024-09-17T07:19:22.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46328]: level=info ts=2024-09-17T07:19:22.458Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=9.347178ms 2024-09-17T07:19:22.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46328]: level=info ts=2024-09-17T07:19:22.460Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2024-09-17T07:19:22.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46328]: level=info ts=2024-09-17T07:19:22.460Z caller=main.go:695 msg="TSDB started" 2024-09-17T07:19:22.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46328]: level=info ts=2024-09-17T07:19:22.460Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T07:19:22.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46328]: level=info ts=2024-09-17T07:19:22.460Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2024-09-17T07:19:22.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46328]: level=info ts=2024-09-17T07:19:22.460Z caller=main.go:561 msg="Stopping notify discovery manager..." 2024-09-17T07:19:22.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46328]: level=info ts=2024-09-17T07:19:22.460Z caller=main.go:583 msg="Stopping scrape manager..." 2024-09-17T07:19:22.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46328]: level=info ts=2024-09-17T07:19:22.460Z caller=main.go:557 msg="Notify discovery manager stopped" 2024-09-17T07:19:22.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46328]: level=info ts=2024-09-17T07:19:22.460Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2024-09-17T07:19:22.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46328]: level=info ts=2024-09-17T07:19:22.460Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2024-09-17T07:19:22.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46328]: level=info ts=2024-09-17T07:19:22.460Z caller=main.go:577 msg="Scrape manager stopped" 2024-09-17T07:19:22.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46328]: level=info ts=2024-09-17T07:19:22.460Z caller=main.go:543 msg="Scrape discovery manager stopped" 2024-09-17T07:19:22.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46328]: level=info ts=2024-09-17T07:19:22.460Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2024-09-17T07:19:22.617 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46328]: level=info ts=2024-09-17T07:19:22.461Z caller=main.go:749 msg="Notifier manager stopped" 2024-09-17T07:19:22.617 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46328]: level=error ts=2024-09-17T07:19:22.461Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 14: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain\n line 31: field http_sd_configs not found in type config.plain\n line 45: field http_sd_configs not found in type config.plain\n line 49: field http_sd_configs not found in type config.plain" 2024-09-17T07:19:22.617 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 bash[46284]: 7068e45188892c0a73ea313f590370a6306b1e31caff10942abc9dc282caa606 2024-09-17T07:19:22.617 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:22 smithi179 podman[46315]: 2024-09-17 07:19:22.486588317 +0000 UTC m=+0.334839075 container died 7068e45188892c0a73ea313f590370a6306b1e31caff10942abc9dc282caa606 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:19:23.317 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 podman[46315]: 2024-09-17 07:19:23.031032382 +0000 UTC m=+0.879283131 container remove 7068e45188892c0a73ea313f590370a6306b1e31caff10942abc9dc282caa606 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:19:23.317 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 bash[46315]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a 2024-09-17T07:19:23.317 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 podman[46370]: 2024-09-17 07:19:23.076347171 +0000 UTC m=+0.017086116 image pull de242295e2257c37c8cadfd962369228f8f10b2d48a44259b65fef44ad4f6490 docker.io/prom/prometheus:v2.18.1 2024-09-17T07:19:23.317 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 podman[46370]: 2024-09-17 07:19:23.190367149 +0000 UTC m=+0.131106146 container create 00447f0fd1bcd6412361a916c840db1acf88fbca7e80cef9d871b923d1527dae (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:19:23.614 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 podman[46370]: 2024-09-17 07:19:23.41717304 +0000 UTC m=+0.357911988 container init 00447f0fd1bcd6412361a916c840db1acf88fbca7e80cef9d871b923d1527dae (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:19:23.614 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 podman[46370]: 2024-09-17 07:19:23.419915711 +0000 UTC m=+0.360654661 container start 00447f0fd1bcd6412361a916c840db1acf88fbca7e80cef9d871b923d1527dae (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:19:23.614 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46384]: level=info ts=2024-09-17T07:19:23.448Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-17T07:19:23.614 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46384]: level=info ts=2024-09-17T07:19:23.448Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2024-09-17T07:19:23.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46384]: level=info ts=2024-09-17T07:19:23.448Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2024-09-17T07:19:23.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46384]: level=info ts=2024-09-17T07:19:23.448Z caller=main.go:339 host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi179 (none))" 2024-09-17T07:19:23.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46384]: level=info ts=2024-09-17T07:19:23.448Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T07:19:23.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46384]: level=info ts=2024-09-17T07:19:23.448Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T07:19:23.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46384]: level=info ts=2024-09-17T07:19:23.449Z caller=main.go:678 msg="Starting TSDB ..." 2024-09-17T07:19:23.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46384]: level=info ts=2024-09-17T07:19:23.449Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2024-09-17T07:19:23.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46384]: level=info ts=2024-09-17T07:19:23.451Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2024-09-17T07:19:23.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46384]: level=info ts=2024-09-17T07:19:23.452Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=7 2024-09-17T07:19:23.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46384]: level=info ts=2024-09-17T07:19:23.459Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=7 2024-09-17T07:19:23.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46384]: level=info ts=2024-09-17T07:19:23.459Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=7 2024-09-17T07:19:23.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46384]: level=info ts=2024-09-17T07:19:23.459Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=7 2024-09-17T07:19:23.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46384]: level=info ts=2024-09-17T07:19:23.459Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=7 2024-09-17T07:19:23.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46384]: level=info ts=2024-09-17T07:19:23.459Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=7 2024-09-17T07:19:23.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46384]: level=info ts=2024-09-17T07:19:23.459Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=7 2024-09-17T07:19:23.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46384]: level=info ts=2024-09-17T07:19:23.459Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=7 maxSegment=7 2024-09-17T07:19:23.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46384]: level=info ts=2024-09-17T07:19:23.459Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=8.128041ms 2024-09-17T07:19:23.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46384]: level=info ts=2024-09-17T07:19:23.461Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2024-09-17T07:19:23.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46384]: level=info ts=2024-09-17T07:19:23.461Z caller=main.go:695 msg="TSDB started" 2024-09-17T07:19:23.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46384]: level=info ts=2024-09-17T07:19:23.461Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T07:19:23.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46384]: level=info ts=2024-09-17T07:19:23.461Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2024-09-17T07:19:23.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46384]: level=info ts=2024-09-17T07:19:23.462Z caller=main.go:561 msg="Stopping notify discovery manager..." 2024-09-17T07:19:23.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46384]: level=info ts=2024-09-17T07:19:23.462Z caller=main.go:583 msg="Stopping scrape manager..." 2024-09-17T07:19:23.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46384]: level=info ts=2024-09-17T07:19:23.462Z caller=main.go:557 msg="Notify discovery manager stopped" 2024-09-17T07:19:23.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46384]: level=info ts=2024-09-17T07:19:23.462Z caller=main.go:543 msg="Scrape discovery manager stopped" 2024-09-17T07:19:23.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46384]: level=info ts=2024-09-17T07:19:23.462Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2024-09-17T07:19:23.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46384]: level=info ts=2024-09-17T07:19:23.462Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2024-09-17T07:19:23.617 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46384]: level=info ts=2024-09-17T07:19:23.462Z caller=main.go:577 msg="Scrape manager stopped" 2024-09-17T07:19:23.617 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46384]: level=info ts=2024-09-17T07:19:23.462Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2024-09-17T07:19:23.617 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46384]: level=info ts=2024-09-17T07:19:23.462Z caller=main.go:749 msg="Notifier manager stopped" 2024-09-17T07:19:23.617 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[46384]: level=error ts=2024-09-17T07:19:23.462Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 14: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain\n line 31: field http_sd_configs not found in type config.plain\n line 45: field http_sd_configs not found in type config.plain\n line 49: field http_sd_configs not found in type config.plain" 2024-09-17T07:19:23.617 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 bash[46370]: 00447f0fd1bcd6412361a916c840db1acf88fbca7e80cef9d871b923d1527dae 2024-09-17T07:19:23.617 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 systemd[1]: Started Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:19:23.617 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:23 smithi179 podman[46401]: 2024-09-17 07:19:23.539110967 +0000 UTC m=+0.069694982 container died 00447f0fd1bcd6412361a916c840db1acf88fbca7e80cef9d871b923d1527dae (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:19:24.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:23 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:22.437671+0000 mgr.y (mgr.25096) 26 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2024-09-17T07:19:24.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:23 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:23.554261+0000 mon.a (mon.0) 771 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:24.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:23 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:23.560539+0000 mon.a (mon.0) 772 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:24.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:23 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:23.564896+0000 mon.c (mon.1) 69 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T07:19:24.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:23 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:23.569193+0000 mon.c (mon.1) 70 : audit [INF] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T07:19:24.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:23 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:23.574355+0000 mon.c (mon.1) 71 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T07:19:24.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:23 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:23.613637+0000 mon.c (mon.1) 72 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:19:24.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:23 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:22.437671+0000 mgr.y (mgr.25096) 26 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2024-09-17T07:19:24.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:23 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:23.554261+0000 mon.a (mon.0) 771 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:24.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:23 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:23.560539+0000 mon.a (mon.0) 772 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:24.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:23 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:23.564896+0000 mon.c (mon.1) 69 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T07:19:24.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:23 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:23.569193+0000 mon.c (mon.1) 70 : audit [INF] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T07:19:24.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:23 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:23.574355+0000 mon.c (mon.1) 71 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T07:19:24.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:23 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:23.613637+0000 mon.c (mon.1) 72 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:19:24.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:22.437671+0000 mgr.y (mgr.25096) 26 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2024-09-17T07:19:24.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:23.554261+0000 mon.a (mon.0) 771 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:24.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:23.560539+0000 mon.a (mon.0) 772 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:24.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:23.564896+0000 mon.c (mon.1) 69 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T07:19:24.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:23.569193+0000 mon.c (mon.1) 70 : audit [INF] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T07:19:24.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:23.574355+0000 mon.c (mon.1) 71 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T07:19:24.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:23.613637+0000 mon.c (mon.1) 72 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:19:24.372 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:24 smithi179 podman[46401]: 2024-09-17 07:19:24.121189929 +0000 UTC m=+0.651773935 container remove 00447f0fd1bcd6412361a916c840db1acf88fbca7e80cef9d871b923d1527dae (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:19:24.372 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:24 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2024-09-17T07:19:24.373 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:24 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Failed with result 'exit-code'. 2024-09-17T07:19:25.043 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:24 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:23.565499+0000 mgr.y (mgr.25096) 27 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T07:19:25.044 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:24 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:23.569556+0000 mgr.y (mgr.25096) 28 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T07:19:25.044 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:24 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:23.574738+0000 mgr.y (mgr.25096) 29 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T07:19:25.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:23.565499+0000 mgr.y (mgr.25096) 27 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T07:19:25.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:23.569556+0000 mgr.y (mgr.25096) 28 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T07:19:25.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:23.574738+0000 mgr.y (mgr.25096) 29 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T07:19:25.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:23.565499+0000 mgr.y (mgr.25096) 27 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T07:19:25.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:23.569556+0000 mgr.y (mgr.25096) 28 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T07:19:25.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:24 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:23.574738+0000 mgr.y (mgr.25096) 29 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T07:19:26.048 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:25 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:24.438561+0000 mgr.y (mgr.25096) 30 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2024-09-17T07:19:26.048 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:25 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:25.392116+0000 mon.c (mon.1) 73 : audit [INF] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T07:19:26.048 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:25 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:25.392635+0000 mon.a (mon.0) 773 : audit [INF] from='mgr.25096 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T07:19:26.048 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:25 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:25.393630+0000 mon.c (mon.1) 74 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T07:19:26.048 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:25 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:25.394482+0000 mon.c (mon.1) 75 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:19:26.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:24.438561+0000 mgr.y (mgr.25096) 30 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2024-09-17T07:19:26.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:25.392116+0000 mon.c (mon.1) 73 : audit [INF] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T07:19:26.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:25.392635+0000 mon.a (mon.0) 773 : audit [INF] from='mgr.25096 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T07:19:26.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:25.393630+0000 mon.c (mon.1) 74 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T07:19:26.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:25.394482+0000 mon.c (mon.1) 75 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:19:26.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:24.438561+0000 mgr.y (mgr.25096) 30 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2024-09-17T07:19:26.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:25.392116+0000 mon.c (mon.1) 73 : audit [INF] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T07:19:26.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:25.392635+0000 mon.a (mon.0) 773 : audit [INF] from='mgr.25096 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T07:19:26.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:25.393630+0000 mon.c (mon.1) 74 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T07:19:26.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:25.394482+0000 mon.c (mon.1) 75 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:19:26.958 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:26 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:19:25.391660+0000 mgr.y (mgr.25096) 31 : cephadm [INF] Upgrade: Updating mgr.x 2024-09-17T07:19:26.958 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:26 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:19:25.395327+0000 mgr.y (mgr.25096) 32 : cephadm [INF] Deploying daemon mgr.x on smithi179 2024-09-17T07:19:27.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:19:25.391660+0000 mgr.y (mgr.25096) 31 : cephadm [INF] Upgrade: Updating mgr.x 2024-09-17T07:19:27.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:19:25.395327+0000 mgr.y (mgr.25096) 32 : cephadm [INF] Deploying daemon mgr.x on smithi179 2024-09-17T07:19:27.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:19:25.391660+0000 mgr.y (mgr.25096) 31 : cephadm [INF] Upgrade: Updating mgr.x 2024-09-17T07:19:27.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:26 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:19:25.395327+0000 mgr.y (mgr.25096) 32 : cephadm [INF] Deploying daemon mgr.x on smithi179 2024-09-17T07:19:27.363 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:26 smithi179 systemd[1]: Stopping Ceph mgr.x for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:19:27.364 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:27 smithi179 podman[46712]: 2024-09-17 07:19:27.161496537 +0000 UTC m=+0.164281861 container died c9b478bf98747d4662e64411af3650acc640bc350c0172e9e5693fd2b0622419 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-17T07:19:27.780 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:27 smithi179 podman[46712]: 2024-09-17 07:19:27.513474729 +0000 UTC m=+0.516260061 container cleanup c9b478bf98747d4662e64411af3650acc640bc350c0172e9e5693fd2b0622419 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-17T07:19:27.780 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:27 smithi179 bash[46712]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x 2024-09-17T07:19:28.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:26.439256+0000 mgr.y (mgr.25096) 33 : cluster [DBG] pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-17T07:19:28.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:26.439256+0000 mgr.y (mgr.25096) 33 : cluster [DBG] pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-17T07:19:28.077 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:27 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:26.439256+0000 mgr.y (mgr.25096) 33 : cluster [DBG] pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-17T07:19:28.077 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:27 smithi179 podman[46725]: 2024-09-17 07:19:27.798193118 +0000 UTC m=+0.640567254 container remove c9b478bf98747d4662e64411af3650acc640bc350c0172e9e5693fd2b0622419 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr.x, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-17T07:19:28.078 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:27 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-17T07:19:28.364 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:28 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-09-17T07:19:28.364 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:28 smithi179 systemd[1]: Stopped Ceph mgr.x for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:19:28.364 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:28 smithi179 systemd[1]: Starting Ceph mgr.x for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:19:28.777 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:28 smithi179 podman[46867]: 2024-09-17 07:19:28.415898747 +0000 UTC m=+0.015795614 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:19:28.777 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:28 smithi179 podman[46867]: 2024-09-17 07:19:28.549095518 +0000 UTC m=+0.148992367 container create 66b52dc26022d87921e090ca1a2e81b19c1175697811cee0089a08e68f661894 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True) 2024-09-17T07:19:29.114 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:28 smithi179 podman[46867]: 2024-09-17 07:19:28.776939806 +0000 UTC m=+0.376836666 container init 66b52dc26022d87921e090ca1a2e81b19c1175697811cee0089a08e68f661894 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True) 2024-09-17T07:19:29.114 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:28 smithi179 podman[46867]: 2024-09-17 07:19:28.77979206 +0000 UTC m=+0.379688908 container start 66b52dc26022d87921e090ca1a2e81b19c1175697811cee0089a08e68f661894 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-17T07:19:29.114 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:28 smithi179 ceph-mgr[46886]: -- 172.21.15.179:0/741600825 <== mon.2 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c58bbdcd00 con 0x55c58bbb1400 2024-09-17T07:19:29.114 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:28 smithi179 bash[46867]: 66b52dc26022d87921e090ca1a2e81b19c1175697811cee0089a08e68f661894 2024-09-17T07:19:29.114 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:28 smithi179 systemd[1]: Started Ceph mgr.x for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:19:29.114 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:28 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:28.899+0000 7ff5efcec040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T07:19:29.114 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:28 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:28.961+0000 7ff5efcec040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T07:19:29.528 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:29.119+0000 7ff5efcec040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T07:19:29.528 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:29.195+0000 7ff5efcec040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T07:19:29.528 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:29.366+0000 7ff5efcec040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T07:19:29.528 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:29.527+0000 7ff5efcec040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T07:19:30.048 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:29.844+0000 7ff5efcec040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T07:19:30.049 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:29.910+0000 7ff5efcec040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T07:19:30.049 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:28.439686+0000 mgr.y (mgr.25096) 34 : cluster [DBG] pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-17T07:19:30.049 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:28.941596+0000 mon.a (mon.0) 774 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:30.049 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:28.954507+0000 mon.a (mon.0) 775 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:30.049 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:28.956602+0000 mon.c (mon.1) 76 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:19:30.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:28.439686+0000 mgr.y (mgr.25096) 34 : cluster [DBG] pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-17T07:19:30.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:28.941596+0000 mon.a (mon.0) 774 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:30.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:28.954507+0000 mon.a (mon.0) 775 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:30.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:28.956602+0000 mon.c (mon.1) 76 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:19:30.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:28.439686+0000 mgr.y (mgr.25096) 34 : cluster [DBG] pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-17T07:19:30.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:28.941596+0000 mon.a (mon.0) 774 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:30.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:28.954507+0000 mon.a (mon.0) 775 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:30.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:29 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:28.956602+0000 mon.c (mon.1) 76 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:19:30.364 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:30 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:30.097+0000 7ff5efcec040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T07:19:31.097 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:31.013+0000 7ff5efcec040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T07:19:31.892 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:30.440380+0000 mgr.y (mgr.25096) 35 : cluster [DBG] pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-17T07:19:31.892 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:30.440380+0000 mgr.y (mgr.25096) 35 : cluster [DBG] pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-17T07:19:32.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:30.440380+0000 mgr.y (mgr.25096) 35 : cluster [DBG] pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-17T07:19:32.114 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:31.096+0000 7ff5efcec040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T07:19:32.114 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:31.890+0000 7ff5efcec040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T07:19:32.114 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:32.015+0000 7ff5efcec040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T07:19:32.433 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:32.163+0000 7ff5efcec040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T07:19:32.434 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:32.223+0000 7ff5efcec040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T07:19:32.434 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:32.375+0000 7ff5efcec040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T07:19:32.727 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:32.444+0000 7ff5efcec040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T07:19:32.727 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:32.503+0000 7ff5efcec040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T07:19:32.728 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:32.663+0000 7ff5efcec040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T07:19:32.728 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:32.726+0000 7ff5efcec040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T07:19:33.015 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:32.657173+0000 mon.c (mon.1) 77 : audit [DBG] from='client.? 172.21.15.44:0/3377458799' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:19:33.015 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:32.801+0000 7ff5efcec040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T07:19:33.015 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:32.892+0000 7ff5efcec040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T07:19:33.015 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:32.956+0000 7ff5efcec040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T07:19:33.015 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:33.014+0000 7ff5efcec040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T07:19:33.044 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:32 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:32.657173+0000 mon.c (mon.1) 77 : audit [DBG] from='client.? 172.21.15.44:0/3377458799' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:19:33.045 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:32 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:32.657173+0000 mon.c (mon.1) 77 : audit [DBG] from='client.? 172.21.15.44:0/3377458799' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:19:34.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:32.440751+0000 mgr.y (mgr.25096) 36 : cluster [DBG] pgmap v18: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:19:34.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:32.998810+0000 mon.a (mon.0) 776 : audit [INF] from='client.? 172.21.15.44:0/89537451' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:6800/2448217567"}]: access denied 2024-09-17T07:19:34.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:32.440751+0000 mgr.y (mgr.25096) 36 : cluster [DBG] pgmap v18: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:19:34.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:32.998810+0000 mon.a (mon.0) 776 : audit [INF] from='client.? 172.21.15.44:0/89537451' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:6800/2448217567"}]: access denied 2024-09-17T07:19:34.114 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:33.906+0000 7ff5efcec040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T07:19:34.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:32.440751+0000 mgr.y (mgr.25096) 36 : cluster [DBG] pgmap v18: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:19:34.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:32.998810+0000 mon.a (mon.0) 776 : audit [INF] from='client.? 172.21.15.44:0/89537451' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:6800/2448217567"}]: access denied 2024-09-17T07:19:34.553 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Scheduled restart job, restart counter is at 1. 2024-09-17T07:19:34.554 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 systemd[1]: Stopped Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:19:34.554 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 systemd[1]: Starting Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:19:34.840 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 podman[47800]: 2024-09-17 07:19:34.45946302 +0000 UTC m=+0.019914101 image pull de242295e2257c37c8cadfd962369228f8f10b2d48a44259b65fef44ad4f6490 docker.io/prom/prometheus:v2.18.1 2024-09-17T07:19:34.841 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 podman[47800]: 2024-09-17 07:19:34.59721182 +0000 UTC m=+0.157662899 container create 0fe04e230fac86f81505055be469c9aa2e379ddf27ea7a68aabad955bbeebc4f (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:19:34.841 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 podman[47800]: 2024-09-17 07:19:34.840820114 +0000 UTC m=+0.401271198 container init 0fe04e230fac86f81505055be469c9aa2e379ddf27ea7a68aabad955bbeebc4f (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:19:34.841 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:33.909837+0000 mon.a (mon.0) 777 : cluster [DBG] Standby manager daemon x restarted 2024-09-17T07:19:34.841 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:33.909957+0000 mon.a (mon.0) 778 : cluster [DBG] Standby manager daemon x started 2024-09-17T07:19:34.841 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:33.910659+0000 mon.b (mon.2) 84 : audit [DBG] from='mgr.? 172.21.15.179:0/1580017420' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T07:19:34.841 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:33.911089+0000 mon.b (mon.2) 85 : audit [DBG] from='mgr.? 172.21.15.179:0/1580017420' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T07:19:34.841 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:33.911841+0000 mon.b (mon.2) 86 : audit [DBG] from='mgr.? 172.21.15.179:0/1580017420' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T07:19:34.841 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:33.912140+0000 mon.b (mon.2) 87 : audit [DBG] from='mgr.? 172.21.15.179:0/1580017420' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T07:19:34.841 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:33.944513+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:34.841 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:33.950325+0000 mon.a (mon.0) 780 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:34.842 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:34.467762+0000 mon.a (mon.0) 781 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:34.842 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:34.473242+0000 mon.a (mon.0) 782 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:35.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:33.909837+0000 mon.a (mon.0) 777 : cluster [DBG] Standby manager daemon x restarted 2024-09-17T07:19:35.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:33.909957+0000 mon.a (mon.0) 778 : cluster [DBG] Standby manager daemon x started 2024-09-17T07:19:35.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:33.910659+0000 mon.b (mon.2) 84 : audit [DBG] from='mgr.? 172.21.15.179:0/1580017420' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T07:19:35.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:33.911089+0000 mon.b (mon.2) 85 : audit [DBG] from='mgr.? 172.21.15.179:0/1580017420' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T07:19:35.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:33.911841+0000 mon.b (mon.2) 86 : audit [DBG] from='mgr.? 172.21.15.179:0/1580017420' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T07:19:35.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:33.912140+0000 mon.b (mon.2) 87 : audit [DBG] from='mgr.? 172.21.15.179:0/1580017420' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T07:19:35.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:33.944513+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:35.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:33.950325+0000 mon.a (mon.0) 780 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:35.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:34.467762+0000 mon.a (mon.0) 781 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:35.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:34.473242+0000 mon.a (mon.0) 782 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:35.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:33.909837+0000 mon.a (mon.0) 777 : cluster [DBG] Standby manager daemon x restarted 2024-09-17T07:19:35.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:33.909957+0000 mon.a (mon.0) 778 : cluster [DBG] Standby manager daemon x started 2024-09-17T07:19:35.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:33.910659+0000 mon.b (mon.2) 84 : audit [DBG] from='mgr.? 172.21.15.179:0/1580017420' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T07:19:35.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:33.911089+0000 mon.b (mon.2) 85 : audit [DBG] from='mgr.? 172.21.15.179:0/1580017420' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T07:19:35.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:33.911841+0000 mon.b (mon.2) 86 : audit [DBG] from='mgr.? 172.21.15.179:0/1580017420' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T07:19:35.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:33.912140+0000 mon.b (mon.2) 87 : audit [DBG] from='mgr.? 172.21.15.179:0/1580017420' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T07:19:35.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:33.944513+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:35.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:33.950325+0000 mon.a (mon.0) 780 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:35.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:34.467762+0000 mon.a (mon.0) 781 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:35.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:34 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:34.473242+0000 mon.a (mon.0) 782 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:35.114 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 podman[47800]: 2024-09-17 07:19:34.844234482 +0000 UTC m=+0.404685563 container start 0fe04e230fac86f81505055be469c9aa2e379ddf27ea7a68aabad955bbeebc4f (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:19:35.114 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[47898]: level=info ts=2024-09-17T07:19:34.874Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-17T07:19:35.114 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[47898]: level=info ts=2024-09-17T07:19:34.874Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2024-09-17T07:19:35.114 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[47898]: level=info ts=2024-09-17T07:19:34.874Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2024-09-17T07:19:35.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[47898]: level=info ts=2024-09-17T07:19:34.874Z caller=main.go:339 host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi179 (none))" 2024-09-17T07:19:35.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[47898]: level=info ts=2024-09-17T07:19:34.874Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T07:19:35.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[47898]: level=info ts=2024-09-17T07:19:34.874Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T07:19:35.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[47898]: level=info ts=2024-09-17T07:19:34.876Z caller=main.go:678 msg="Starting TSDB ..." 2024-09-17T07:19:35.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[47898]: level=info ts=2024-09-17T07:19:34.876Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2024-09-17T07:19:35.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[47898]: level=info ts=2024-09-17T07:19:34.878Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2024-09-17T07:19:35.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[47898]: level=info ts=2024-09-17T07:19:34.878Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=8 2024-09-17T07:19:35.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[47898]: level=info ts=2024-09-17T07:19:34.885Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=8 2024-09-17T07:19:35.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[47898]: level=info ts=2024-09-17T07:19:34.885Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=8 2024-09-17T07:19:35.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[47898]: level=info ts=2024-09-17T07:19:34.885Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=8 2024-09-17T07:19:35.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[47898]: level=info ts=2024-09-17T07:19:34.885Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=8 2024-09-17T07:19:35.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[47898]: level=info ts=2024-09-17T07:19:34.885Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=8 2024-09-17T07:19:35.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[47898]: level=info ts=2024-09-17T07:19:34.885Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=8 2024-09-17T07:19:35.116 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[47898]: level=info ts=2024-09-17T07:19:34.886Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=7 maxSegment=8 2024-09-17T07:19:35.116 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[47898]: level=info ts=2024-09-17T07:19:34.886Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=8 maxSegment=8 2024-09-17T07:19:35.116 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[47898]: level=info ts=2024-09-17T07:19:34.886Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=8.094772ms 2024-09-17T07:19:35.116 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[47898]: level=info ts=2024-09-17T07:19:34.887Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2024-09-17T07:19:35.116 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[47898]: level=info ts=2024-09-17T07:19:34.888Z caller=main.go:695 msg="TSDB started" 2024-09-17T07:19:35.116 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[47898]: level=info ts=2024-09-17T07:19:34.888Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T07:19:35.116 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[47898]: level=info ts=2024-09-17T07:19:34.888Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2024-09-17T07:19:35.116 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[47898]: level=info ts=2024-09-17T07:19:34.888Z caller=main.go:561 msg="Stopping notify discovery manager..." 2024-09-17T07:19:35.116 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[47898]: level=info ts=2024-09-17T07:19:34.888Z caller=main.go:583 msg="Stopping scrape manager..." 2024-09-17T07:19:35.116 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[47898]: level=info ts=2024-09-17T07:19:34.888Z caller=main.go:557 msg="Notify discovery manager stopped" 2024-09-17T07:19:35.116 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[47898]: level=info ts=2024-09-17T07:19:34.888Z caller=main.go:543 msg="Scrape discovery manager stopped" 2024-09-17T07:19:35.116 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[47898]: level=info ts=2024-09-17T07:19:34.888Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2024-09-17T07:19:35.116 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[47898]: level=info ts=2024-09-17T07:19:34.888Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2024-09-17T07:19:35.116 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[47898]: level=info ts=2024-09-17T07:19:34.888Z caller=main.go:577 msg="Scrape manager stopped" 2024-09-17T07:19:35.117 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[47898]: level=info ts=2024-09-17T07:19:34.888Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2024-09-17T07:19:35.117 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[47898]: level=info ts=2024-09-17T07:19:34.888Z caller=main.go:749 msg="Notifier manager stopped" 2024-09-17T07:19:35.117 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[47898]: level=error ts=2024-09-17T07:19:34.888Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 14: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain\n line 31: field http_sd_configs not found in type config.plain\n line 45: field http_sd_configs not found in type config.plain\n line 49: field http_sd_configs not found in type config.plain" 2024-09-17T07:19:35.117 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 bash[47800]: 0fe04e230fac86f81505055be469c9aa2e379ddf27ea7a68aabad955bbeebc4f 2024-09-17T07:19:35.117 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 systemd[1]: Started Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:19:35.117 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:34 smithi179 podman[47930]: 2024-09-17 07:19:34.930981059 +0000 UTC m=+0.034876447 container died 0fe04e230fac86f81505055be469c9aa2e379ddf27ea7a68aabad955bbeebc4f (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:19:35.720 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:35 smithi179 podman[47930]: 2024-09-17 07:19:35.461273773 +0000 UTC m=+0.565169197 container remove 0fe04e230fac86f81505055be469c9aa2e379ddf27ea7a68aabad955bbeebc4f (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:19:35.721 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:35 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2024-09-17T07:19:35.721 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:35 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Failed with result 'exit-code'. 2024-09-17T07:19:35.981 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:35 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:34.441513+0000 mgr.y (mgr.25096) 37 : cluster [DBG] pgmap v19: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-17T07:19:35.981 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:35 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:34.808878+0000 mon.a (mon.0) 783 : cluster [DBG] mgrmap e24: y(active, since 33s), standbys: x 2024-09-17T07:19:36.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:34.441513+0000 mgr.y (mgr.25096) 37 : cluster [DBG] pgmap v19: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-17T07:19:36.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:34.808878+0000 mon.a (mon.0) 783 : cluster [DBG] mgrmap e24: y(active, since 33s), standbys: x 2024-09-17T07:19:36.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:34.441513+0000 mgr.y (mgr.25096) 37 : cluster [DBG] pgmap v19: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-17T07:19:36.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:35 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:34.808878+0000 mon.a (mon.0) 783 : cluster [DBG] mgrmap e24: y(active, since 33s), standbys: x 2024-09-17T07:19:37.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:36.441862+0000 mgr.y (mgr.25096) 38 : cluster [DBG] pgmap v20: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:19:37.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:36.518246+0000 mon.a (mon.0) 784 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:37.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:36.523365+0000 mon.a (mon.0) 785 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:37.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:36.441862+0000 mgr.y (mgr.25096) 38 : cluster [DBG] pgmap v20: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:19:37.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:36.518246+0000 mon.a (mon.0) 784 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:37.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:37 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:36.523365+0000 mon.a (mon.0) 785 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:37.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:37 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:36.441862+0000 mgr.y (mgr.25096) 38 : cluster [DBG] pgmap v20: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:19:37.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:37 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:36.518246+0000 mon.a (mon.0) 784 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:37.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:37 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:36.523365+0000 mon.a (mon.0) 785 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:39.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:38.263448+0000 mon.a (mon.0) 786 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:39.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:38.272102+0000 mon.a (mon.0) 787 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:39.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:38.273883+0000 mon.c (mon.1) 78 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:19:39.265 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:38.274882+0000 mon.c (mon.1) 79 : audit [INF] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:19:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:19:38.275712+0000 mgr.y (mgr.25096) 39 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:19:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:38.276354+0000 mgr.y (mgr.25096) 40 : cluster [DBG] pgmap v21: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2024-09-17T07:19:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:38.276628+0000 mgr.y (mgr.25096) 41 : cluster [DBG] pgmap v22: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2024-09-17T07:19:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:38.283417+0000 mon.a (mon.0) 788 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:38.403189+0000 mon.c (mon.1) 80 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:19:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:38.406201+0000 mon.c (mon.1) 81 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:19:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:19:38.406917+0000 mgr.y (mgr.25096) 42 : cephadm [INF] Upgrade: Setting container_image for all mgr 2024-09-17T07:19:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:38.411841+0000 mon.a (mon.0) 789 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:38.413142+0000 mon.c (mon.1) 82 : audit [INF] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-17T07:19:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:38.413420+0000 mon.a (mon.0) 790 : audit [INF] from='mgr.25096 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-17T07:19:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:38.418431+0000 mon.a (mon.0) 791 : audit [INF] from='mgr.25096 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-17T07:19:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:38.419415+0000 mon.c (mon.1) 83 : audit [INF] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-17T07:19:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:38.419685+0000 mon.a (mon.0) 792 : audit [INF] from='mgr.25096 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-17T07:19:39.266 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:38.424657+0000 mon.a (mon.0) 793 : audit [INF] from='mgr.25096 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-17T07:19:39.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:38.425961+0000 mon.c (mon.1) 84 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T07:19:39.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:38.426618+0000 mon.c (mon.1) 85 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-17T07:19:39.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:19:38.427079+0000 mgr.y (mgr.25096) 43 : cephadm [INF] Upgrade: It appears safe to stop mon.a 2024-09-17T07:19:39.267 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:38.263448+0000 mon.a (mon.0) 786 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:39.267 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:38.272102+0000 mon.a (mon.0) 787 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:39.267 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:38.273883+0000 mon.c (mon.1) 78 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:19:39.267 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:38.274882+0000 mon.c (mon.1) 79 : audit [INF] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:19:39.267 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:19:38.275712+0000 mgr.y (mgr.25096) 39 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:19:39.267 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:38.276354+0000 mgr.y (mgr.25096) 40 : cluster [DBG] pgmap v21: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2024-09-17T07:19:39.267 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:38.276628+0000 mgr.y (mgr.25096) 41 : cluster [DBG] pgmap v22: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2024-09-17T07:19:39.267 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:38.283417+0000 mon.a (mon.0) 788 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:39.268 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:38.403189+0000 mon.c (mon.1) 80 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:19:39.268 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:38.406201+0000 mon.c (mon.1) 81 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:19:39.268 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:19:38.406917+0000 mgr.y (mgr.25096) 42 : cephadm [INF] Upgrade: Setting container_image for all mgr 2024-09-17T07:19:39.268 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:38.411841+0000 mon.a (mon.0) 789 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:39.268 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:38.413142+0000 mon.c (mon.1) 82 : audit [INF] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-17T07:19:39.268 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:38.413420+0000 mon.a (mon.0) 790 : audit [INF] from='mgr.25096 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-17T07:19:39.268 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:38.418431+0000 mon.a (mon.0) 791 : audit [INF] from='mgr.25096 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-17T07:19:39.268 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:38.419415+0000 mon.c (mon.1) 83 : audit [INF] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-17T07:19:39.268 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:38.419685+0000 mon.a (mon.0) 792 : audit [INF] from='mgr.25096 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-17T07:19:39.268 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:38.424657+0000 mon.a (mon.0) 793 : audit [INF] from='mgr.25096 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-17T07:19:39.268 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:38.425961+0000 mon.c (mon.1) 84 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T07:19:39.268 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:38.426618+0000 mon.c (mon.1) 85 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-17T07:19:39.268 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:39 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:19:38.427079+0000 mgr.y (mgr.25096) 43 : cephadm [INF] Upgrade: It appears safe to stop mon.a 2024-09-17T07:19:39.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:38.263448+0000 mon.a (mon.0) 786 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:39.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:38.272102+0000 mon.a (mon.0) 787 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:39.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:38.273883+0000 mon.c (mon.1) 78 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:19:39.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:38.274882+0000 mon.c (mon.1) 79 : audit [INF] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:19:39.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:19:38.275712+0000 mgr.y (mgr.25096) 39 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:19:39.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:38.276354+0000 mgr.y (mgr.25096) 40 : cluster [DBG] pgmap v21: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2024-09-17T07:19:39.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:38.276628+0000 mgr.y (mgr.25096) 41 : cluster [DBG] pgmap v22: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2024-09-17T07:19:39.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:38.283417+0000 mon.a (mon.0) 788 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:39.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:38.403189+0000 mon.c (mon.1) 80 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:19:39.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:38.406201+0000 mon.c (mon.1) 81 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:19:39.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:19:38.406917+0000 mgr.y (mgr.25096) 42 : cephadm [INF] Upgrade: Setting container_image for all mgr 2024-09-17T07:19:39.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:38.411841+0000 mon.a (mon.0) 789 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:39.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:38.413142+0000 mon.c (mon.1) 82 : audit [INF] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-17T07:19:39.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:38.413420+0000 mon.a (mon.0) 790 : audit [INF] from='mgr.25096 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-17T07:19:39.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:38.418431+0000 mon.a (mon.0) 791 : audit [INF] from='mgr.25096 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-17T07:19:39.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:38.419415+0000 mon.c (mon.1) 83 : audit [INF] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-17T07:19:39.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:38.419685+0000 mon.a (mon.0) 792 : audit [INF] from='mgr.25096 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-17T07:19:39.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:38.424657+0000 mon.a (mon.0) 793 : audit [INF] from='mgr.25096 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-17T07:19:39.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:38.425961+0000 mon.c (mon.1) 84 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T07:19:39.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:38.426618+0000 mon.c (mon.1) 85 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-17T07:19:39.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:19:38.427079+0000 mgr.y (mgr.25096) 43 : cephadm [INF] Upgrade: It appears safe to stop mon.a 2024-09-17T07:19:41.234 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:19:39.936067+0000 mgr.y (mgr.25096) 44 : cephadm [INF] Upgrade: Updating mon.a 2024-09-17T07:19:41.234 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:39.941338+0000 mon.a (mon.0) 794 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:41.234 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:39.942643+0000 mon.c (mon.1) 86 : audit [INF] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T07:19:41.234 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:39.943347+0000 mon.c (mon.1) 87 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T07:19:41.234 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: audit 2024-09-17T07:19:39.943993+0000 mon.c (mon.1) 88 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:19:41.234 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cephadm 2024-09-17T07:19:39.944979+0000 mgr.y (mgr.25096) 45 : cephadm [INF] Deploying daemon mon.a on smithi044 2024-09-17T07:19:41.234 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: cluster 2024-09-17T07:19:40.277313+0000 mgr.y (mgr.25096) 46 : cluster [DBG] pgmap v23: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2024-09-17T07:19:41.234 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:19:39.936067+0000 mgr.y (mgr.25096) 44 : cephadm [INF] Upgrade: Updating mon.a 2024-09-17T07:19:41.234 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:39.941338+0000 mon.a (mon.0) 794 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:41.235 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:39.942643+0000 mon.c (mon.1) 86 : audit [INF] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T07:19:41.235 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:39.943347+0000 mon.c (mon.1) 87 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T07:19:41.235 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:39.943993+0000 mon.c (mon.1) 88 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:19:41.235 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:19:39.944979+0000 mgr.y (mgr.25096) 45 : cephadm [INF] Deploying daemon mon.a on smithi044 2024-09-17T07:19:41.235 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:40 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:40.277313+0000 mgr.y (mgr.25096) 46 : cluster [DBG] pgmap v23: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2024-09-17T07:19:41.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:40 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:19:39.936067+0000 mgr.y (mgr.25096) 44 : cephadm [INF] Upgrade: Updating mon.a 2024-09-17T07:19:41.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:40 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:39.941338+0000 mon.a (mon.0) 794 : audit [INF] from='mgr.25096 ' entity='mgr.y' 2024-09-17T07:19:41.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:40 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:39.942643+0000 mon.c (mon.1) 86 : audit [INF] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T07:19:41.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:40 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:39.943347+0000 mon.c (mon.1) 87 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T07:19:41.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:40 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:39.943993+0000 mon.c (mon.1) 88 : audit [DBG] from='mgr.25096 172.21.15.44:0/507713802' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:19:41.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:40 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:19:39.944979+0000 mgr.y (mgr.25096) 45 : cephadm [INF] Deploying daemon mon.a on smithi044 2024-09-17T07:19:41.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:40 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:40.277313+0000 mgr.y (mgr.25096) 46 : cluster [DBG] pgmap v23: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2024-09-17T07:19:41.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:41 smithi044 systemd[1]: Stopping Ceph mon.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:19:41.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:41 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:19:41.684+0000 7f713a68c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-09-17T07:19:41.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:41 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a[23634]: debug 2024-09-17T07:19:41.684+0000 7f713a68c700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-17T07:19:41.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:41 smithi044 podman[58018]: 2024-09-17 07:19:41.735870286 +0000 UTC m=+0.162732918 container died 7905efa25a24d56dd5270ce48fb2ec6b71b09f25f2272ad1322ce65bb8a3ebaf (image=quay.io/ceph/ceph:v16.2.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, org.label-schema.build-date=20201204, GIT_CLEAN=True, maintainer=Dimitri Savineau , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-16.2.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image, RELEASE=HEAD, ceph=True) 2024-09-17T07:19:42.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:42 smithi044 podman[58018]: 2024-09-17 07:19:42.112805852 +0000 UTC m=+0.539668486 container cleanup 7905efa25a24d56dd5270ce48fb2ec6b71b09f25f2272ad1322ce65bb8a3ebaf (image=quay.io/ceph/ceph:v16.2.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a, org.label-schema.name=CentOS Base Image, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Dimitri Savineau , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-16.2.0, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, ceph=True, org.label-schema.build-date=20201204, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-17T07:19:42.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:42 smithi044 bash[58018]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a 2024-09-17T07:19:42.653 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:42 smithi044 podman[58029]: 2024-09-17 07:19:42.389671033 +0000 UTC m=+0.655391838 container remove 7905efa25a24d56dd5270ce48fb2ec6b71b09f25f2272ad1322ce65bb8a3ebaf (image=quay.io/ceph/ceph:v16.2.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.a, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Dimitri Savineau , org.label-schema.build-date=20201204, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-16.2.0, org.label-schema.name=CentOS Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, RELEASE=HEAD, org.label-schema.license=GPLv2) 2024-09-17T07:19:42.653 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:42 smithi044 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-17T07:19:42.653 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:42 smithi044 systemd[1]: Stopped Ceph mon.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:19:43.068 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:42 smithi044 systemd[1]: Starting Ceph mon.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:19:43.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 podman[58172]: 2024-09-17 07:19:42.970850209 +0000 UTC m=+0.016280671 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:19:43.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 podman[58172]: 2024-09-17 07:19:43.09902551 +0000 UTC m=+0.144455971 container create c3293e8cdbbafc442e96bf50af8c9ceec22396bae67d61f59e1f5bae54ba9f2e (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon-a, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T07:19:43.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 podman[58172]: 2024-09-17 07:19:43.368764552 +0000 UTC m=+0.414195013 container init c3293e8cdbbafc442e96bf50af8c9ceec22396bae67d61f59e1f5bae54ba9f2e (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-17T07:19:43.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 podman[58172]: 2024-09-17 07:19:43.372632173 +0000 UTC m=+0.418062635 container start c3293e8cdbbafc442e96bf50af8c9ceec22396bae67d61f59e1f5bae54ba9f2e (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon-a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-17T07:19:43.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: set uid:gid to 167:167 (ceph:ceph) 2024-09-17T07:19:43.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable), process ceph-mon, pid 2 2024-09-17T07:19:43.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: pidfile_write: ignore empty --pid-file 2024-09-17T07:19:43.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: load: jerasure load: lrc 2024-09-17T07:19:43.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: RocksDB version: 7.9.2 2024-09-17T07:19:43.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Git sha 0 2024-09-17T07:19:43.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Compile date 2024-09-16 15:15:13 2024-09-17T07:19:43.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: DB SUMMARY 2024-09-17T07:19:43.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: DB Session ID: SVZRBJSNCMQI2DEA97E9 2024-09-17T07:19:43.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: CURRENT file: CURRENT 2024-09-17T07:19:43.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: IDENTITY file: IDENTITY 2024-09-17T07:19:43.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: MANIFEST file: MANIFEST-000008 size: 972 Bytes 2024-09-17T07:19:43.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000023.sst 2024-09-17T07:19:43.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000021.log size: 1189502 ; 2024-09-17T07:19:43.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.error_if_exists: 0 2024-09-17T07:19:43.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.create_if_missing: 0 2024-09-17T07:19:43.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.paranoid_checks: 1 2024-09-17T07:19:43.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-17T07:19:43.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-17T07:19:43.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-17T07:19:43.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.env: 0x5564219debc0 2024-09-17T07:19:43.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.fs: PosixFileSystem 2024-09-17T07:19:43.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.info_log: 0x5564232d8be0 2024-09-17T07:19:43.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.max_file_opening_threads: 16 2024-09-17T07:19:43.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.statistics: (nil) 2024-09-17T07:19:43.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.use_fsync: 0 2024-09-17T07:19:43.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.max_log_file_size: 0 2024-09-17T07:19:43.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-17T07:19:43.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-17T07:19:43.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.keep_log_file_num: 1000 2024-09-17T07:19:43.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.recycle_log_file_num: 0 2024-09-17T07:19:43.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.allow_fallocate: 1 2024-09-17T07:19:43.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.allow_mmap_reads: 0 2024-09-17T07:19:43.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.allow_mmap_writes: 0 2024-09-17T07:19:43.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.use_direct_reads: 0 2024-09-17T07:19:43.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-17T07:19:43.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.create_missing_column_families: 0 2024-09-17T07:19:43.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.db_log_dir: 2024-09-17T07:19:43.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.wal_dir: 2024-09-17T07:19:43.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-17T07:19:43.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-17T07:19:43.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-17T07:19:43.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-17T07:19:43.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-17T07:19:43.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-17T07:19:43.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.advise_random_on_open: 1 2024-09-17T07:19:43.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.db_write_buffer_size: 0 2024-09-17T07:19:43.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.write_buffer_manager: 0x5564232e6b40 2024-09-17T07:19:43.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-17T07:19:43.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-17T07:19:43.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-17T07:19:43.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.rate_limiter: (nil) 2024-09-17T07:19:43.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-17T07:19:43.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.wal_recovery_mode: 2 2024-09-17T07:19:43.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.enable_thread_tracking: 0 2024-09-17T07:19:43.830 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.enable_pipelined_write: 0 2024-09-17T07:19:43.830 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.unordered_write: 0 2024-09-17T07:19:43.830 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-17T07:19:43.830 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-17T07:19:43.830 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-17T07:19:43.830 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-17T07:19:43.830 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.row_cache: None 2024-09-17T07:19:43.830 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.wal_filter: None 2024-09-17T07:19:43.830 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-17T07:19:43.835 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.allow_ingest_behind: 0 2024-09-17T07:19:43.835 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.two_write_queues: 0 2024-09-17T07:19:43.835 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.manual_wal_flush: 0 2024-09-17T07:19:43.835 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.wal_compression: 0 2024-09-17T07:19:43.835 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.atomic_flush: 0 2024-09-17T07:19:43.835 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-17T07:19:43.835 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-17T07:19:43.835 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-17T07:19:43.835 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.log_readahead_size: 0 2024-09-17T07:19:43.836 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-17T07:19:43.836 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.best_efforts_recovery: 0 2024-09-17T07:19:43.836 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-17T07:19:43.836 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-17T07:19:43.836 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.allow_data_in_errors: 0 2024-09-17T07:19:43.836 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.db_host_id: __hostname__ 2024-09-17T07:19:43.836 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-17T07:19:43.836 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.max_background_jobs: 2 2024-09-17T07:19:43.836 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.max_background_compactions: -1 2024-09-17T07:19:43.836 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.max_subcompactions: 1 2024-09-17T07:19:43.836 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-17T07:19:43.836 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-17T07:19:43.837 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-17T07:19:43.837 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.max_total_wal_size: 0 2024-09-17T07:19:43.837 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-17T07:19:43.837 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-17T07:19:43.837 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-17T07:19:43.837 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-17T07:19:43.837 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.max_open_files: -1 2024-09-17T07:19:43.837 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.bytes_per_sync: 0 2024-09-17T07:19:43.837 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-17T07:19:43.837 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-17T07:19:43.837 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.compaction_readahead_size: 0 2024-09-17T07:19:43.837 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.max_background_flushes: -1 2024-09-17T07:19:43.837 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Compression algorithms supported: 2024-09-17T07:19:43.837 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: kZSTD supported: 0 2024-09-17T07:19:43.838 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: kXpressCompression supported: 0 2024-09-17T07:19:43.838 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: kBZip2Compression supported: 0 2024-09-17T07:19:43.838 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-17T07:19:43.838 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: kLZ4Compression supported: 1 2024-09-17T07:19:43.838 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: kZlibCompression supported: 1 2024-09-17T07:19:43.838 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: kLZ4HCCompression supported: 1 2024-09-17T07:19:43.838 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: kSnappyCompression supported: 1 2024-09-17T07:19:43.838 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-17T07:19:43.838 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-17T07:19:43.838 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000008 2024-09-17T07:19:43.838 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-17T07:19:43.838 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-17T07:19:43.838 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.merge_operator: 2024-09-17T07:19:43.839 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.compaction_filter: None 2024-09-17T07:19:43.839 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.compaction_filter_factory: None 2024-09-17T07:19:43.839 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.sst_partitioner_factory: None 2024-09-17T07:19:43.839 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-17T07:19:43.839 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-17T07:19:43.839 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5564232d87e0) 2024-09-17T07:19:43.839 INFO:journalctl@ceph.mon.a.smithi044.stdout: cache_index_and_filter_blocks: 1 2024-09-17T07:19:43.839 INFO:journalctl@ceph.mon.a.smithi044.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-17T07:19:43.839 INFO:journalctl@ceph.mon.a.smithi044.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-17T07:19:43.839 INFO:journalctl@ceph.mon.a.smithi044.stdout: pin_top_level_index_and_filter: 1 2024-09-17T07:19:43.839 INFO:journalctl@ceph.mon.a.smithi044.stdout: index_type: 0 2024-09-17T07:19:43.839 INFO:journalctl@ceph.mon.a.smithi044.stdout: data_block_index_type: 0 2024-09-17T07:19:43.839 INFO:journalctl@ceph.mon.a.smithi044.stdout: index_shortening: 1 2024-09-17T07:19:43.840 INFO:journalctl@ceph.mon.a.smithi044.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-17T07:19:43.840 INFO:journalctl@ceph.mon.a.smithi044.stdout: checksum: 4 2024-09-17T07:19:43.840 INFO:journalctl@ceph.mon.a.smithi044.stdout: no_block_cache: 0 2024-09-17T07:19:43.840 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_cache: 0x5564232cd350 2024-09-17T07:19:43.840 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_cache_name: BinnedLRUCache 2024-09-17T07:19:43.840 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_cache_options: 2024-09-17T07:19:43.840 INFO:journalctl@ceph.mon.a.smithi044.stdout: capacity : 536870912 2024-09-17T07:19:43.840 INFO:journalctl@ceph.mon.a.smithi044.stdout: num_shard_bits : 4 2024-09-17T07:19:43.840 INFO:journalctl@ceph.mon.a.smithi044.stdout: strict_capacity_limit : 0 2024-09-17T07:19:43.840 INFO:journalctl@ceph.mon.a.smithi044.stdout: high_pri_pool_ratio: 0.000 2024-09-17T07:19:43.840 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_cache_compressed: (nil) 2024-09-17T07:19:43.840 INFO:journalctl@ceph.mon.a.smithi044.stdout: persistent_cache: (nil) 2024-09-17T07:19:43.840 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_size: 4096 2024-09-17T07:19:43.841 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_size_deviation: 10 2024-09-17T07:19:43.841 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_restart_interval: 16 2024-09-17T07:19:43.841 INFO:journalctl@ceph.mon.a.smithi044.stdout: index_block_restart_interval: 1 2024-09-17T07:19:43.841 INFO:journalctl@ceph.mon.a.smithi044.stdout: metadata_block_size: 4096 2024-09-17T07:19:43.841 INFO:journalctl@ceph.mon.a.smithi044.stdout: partition_filters: 0 2024-09-17T07:19:43.841 INFO:journalctl@ceph.mon.a.smithi044.stdout: use_delta_encoding: 1 2024-09-17T07:19:43.841 INFO:journalctl@ceph.mon.a.smithi044.stdout: filter_policy: bloomfilter 2024-09-17T07:19:43.841 INFO:journalctl@ceph.mon.a.smithi044.stdout: whole_key_filtering: 1 2024-09-17T07:19:43.841 INFO:journalctl@ceph.mon.a.smithi044.stdout: verify_compression: 0 2024-09-17T07:19:43.841 INFO:journalctl@ceph.mon.a.smithi044.stdout: read_amp_bytes_per_bit: 0 2024-09-17T07:19:43.841 INFO:journalctl@ceph.mon.a.smithi044.stdout: format_version: 5 2024-09-17T07:19:43.841 INFO:journalctl@ceph.mon.a.smithi044.stdout: enable_index_compression: 1 2024-09-17T07:19:43.841 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_align: 0 2024-09-17T07:19:43.842 INFO:journalctl@ceph.mon.a.smithi044.stdout: max_auto_readahead_size: 262144 2024-09-17T07:19:43.842 INFO:journalctl@ceph.mon.a.smithi044.stdout: prepopulate_block_cache: 0 2024-09-17T07:19:43.842 INFO:journalctl@ceph.mon.a.smithi044.stdout: initial_auto_readahead_size: 8192 2024-09-17T07:19:43.842 INFO:journalctl@ceph.mon.a.smithi044.stdout: num_file_reads_for_auto_readahead: 2 2024-09-17T07:19:43.842 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.write_buffer_size: 33554432 2024-09-17T07:19:43.842 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.max_write_buffer_number: 2 2024-09-17T07:19:43.842 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.compression: NoCompression 2024-09-17T07:19:43.842 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.bottommost_compression: Disabled 2024-09-17T07:19:43.842 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.prefix_extractor: nullptr 2024-09-17T07:19:43.842 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-17T07:19:43.842 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.num_levels: 7 2024-09-17T07:19:43.842 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-17T07:19:43.842 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-17T07:19:43.843 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-17T07:19:43.843 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-17T07:19:43.843 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-17T07:19:43.843 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-17T07:19:43.843 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-17T07:19:43.843 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-17T07:19:43.843 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-17T07:19:43.843 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-17T07:19:43.843 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-17T07:19:43.843 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-17T07:19:43.843 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-17T07:19:43.843 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.compression_opts.level: 32767 2024-09-17T07:19:43.843 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.compression_opts.strategy: 0 2024-09-17T07:19:43.843 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-17T07:19:43.844 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-17T07:19:43.844 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-17T07:19:43.844 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-17T07:19:43.844 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.compression_opts.enabled: false 2024-09-17T07:19:43.844 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-17T07:19:43.844 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-17T07:19:43.844 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-17T07:19:43.844 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-17T07:19:43.844 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.target_file_size_base: 67108864 2024-09-17T07:19:43.844 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-17T07:19:43.844 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-17T07:19:43.844 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-17T07:19:43.844 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-17T07:19:43.845 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-17T07:19:43.845 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-17T07:19:43.845 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-17T07:19:43.845 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-17T07:19:43.845 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-17T07:19:43.845 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-17T07:19:43.845 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-17T07:19:43.845 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-17T07:19:43.845 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-17T07:19:43.845 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-17T07:19:43.845 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.arena_block_size: 1048576 2024-09-17T07:19:43.845 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-17T07:19:43.845 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-17T07:19:43.845 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.disable_auto_compactions: 0 2024-09-17T07:19:43.846 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-17T07:19:43.846 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-17T07:19:43.846 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-17T07:19:43.846 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-17T07:19:43.846 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-17T07:19:43.846 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-17T07:19:43.846 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-17T07:19:43.846 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-17T07:19:43.846 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-17T07:19:43.846 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-17T07:19:43.846 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-17T07:19:43.846 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.inplace_update_support: 0 2024-09-17T07:19:43.846 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-17T07:19:43.847 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-17T07:19:43.847 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-17T07:19:43.847 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-17T07:19:43.847 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.bloom_locality: 0 2024-09-17T07:19:43.847 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.max_successive_merges: 0 2024-09-17T07:19:43.847 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-17T07:19:43.847 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.paranoid_file_checks: 0 2024-09-17T07:19:43.847 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.force_consistency_checks: 1 2024-09-17T07:19:43.847 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.report_bg_io_stats: 0 2024-09-17T07:19:43.847 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.ttl: 2592000 2024-09-17T07:19:43.847 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-17T07:19:43.847 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-17T07:19:43.847 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-17T07:19:43.847 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.enable_blob_files: false 2024-09-17T07:19:43.848 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.min_blob_size: 0 2024-09-17T07:19:43.848 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.blob_file_size: 268435456 2024-09-17T07:19:43.848 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-17T07:19:43.848 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-17T07:19:43.848 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-17T07:19:43.848 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-17T07:19:43.848 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-17T07:19:43.848 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.blob_file_starting_level: 0 2024-09-17T07:19:43.848 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-17T07:19:43.848 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 23.sst 2024-09-17T07:19:43.848 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-17T07:19:43.848 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000008 succeeded,manifest_file_number is 8, next_file_number is 25, last_sequence is 11301, log_number is 21,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-17T07:19:43.849 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 21 2024-09-17T07:19:43.849 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: bd2df17e-4141-477f-9c94-cad6cc87d7ca 2024-09-17T07:19:43.849 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726557583399920, "job": 1, "event": "recovery_started", "wal_files": [21]} 2024-09-17T07:19:43.849 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #21 mode 2 2024-09-17T07:19:43.849 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726557583404140, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 26, "file_size": 1150627, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11295, "largest_seqno": 11557, "table_properties": {"data_size": 1148108, "index_size": 1161, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 325, "raw_key_size": 2610, "raw_average_key_size": 24, "raw_value_size": 1145052, "raw_average_value_size": 10602, "num_data_blocks": 51, "num_entries": 108, "num_filter_entries": 108, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726557583, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "bd2df17e-4141-477f-9c94-cad6cc87d7ca", "db_session_id": "SVZRBJSNCMQI2DEA97E9", "orig_file_number": 26, "seqno_to_time_mapping": "N/A"}} 2024-09-17T07:19:43.849 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726557583404249, "job": 1, "event": "recovery_finished"} 2024-09-17T07:19:43.849 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: [db/version_set.cc:5047] Creating manifest 28 2024-09-17T07:19:43.849 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-17T07:19:43.849 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000021.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-17T07:19:43.849 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5564232f8e00 2024-09-17T07:19:43.849 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: DB pointer 0x5564233ec000 2024-09-17T07:19:43.849 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-17T07:19:43.849 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-17T07:19:43.850 INFO:journalctl@ceph.mon.a.smithi044.stdout: ** DB Stats ** 2024-09-17T07:19:43.850 INFO:journalctl@ceph.mon.a.smithi044.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T07:19:43.850 INFO:journalctl@ceph.mon.a.smithi044.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-17T07:19:43.850 INFO:journalctl@ceph.mon.a.smithi044.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T07:19:43.850 INFO:journalctl@ceph.mon.a.smithi044.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T07:19:43.850 INFO:journalctl@ceph.mon.a.smithi044.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-17T07:19:43.850 INFO:journalctl@ceph.mon.a.smithi044.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T07:19:43.850 INFO:journalctl@ceph.mon.a.smithi044.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T07:19:43.850 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2024-09-17T07:19:43.850 INFO:journalctl@ceph.mon.a.smithi044.stdout: ** Compaction Stats [default] ** 2024-09-17T07:19:43.850 INFO:journalctl@ceph.mon.a.smithi044.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T07:19:43.850 INFO:journalctl@ceph.mon.a.smithi044.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-17T07:19:43.850 INFO:journalctl@ceph.mon.a.smithi044.stdout: L0 1/0 1.10 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 436.3 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-17T07:19:43.851 INFO:journalctl@ceph.mon.a.smithi044.stdout: L6 1/0 24.60 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-17T07:19:43.851 INFO:journalctl@ceph.mon.a.smithi044.stdout: Sum 2/0 25.70 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 436.3 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-17T07:19:43.851 INFO:journalctl@ceph.mon.a.smithi044.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 436.3 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-17T07:19:43.851 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2024-09-17T07:19:43.851 INFO:journalctl@ceph.mon.a.smithi044.stdout: ** Compaction Stats [default] ** 2024-09-17T07:19:43.851 INFO:journalctl@ceph.mon.a.smithi044.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T07:19:43.851 INFO:journalctl@ceph.mon.a.smithi044.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T07:19:43.851 INFO:journalctl@ceph.mon.a.smithi044.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 436.3 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-17T07:19:43.851 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2024-09-17T07:19:43.851 INFO:journalctl@ceph.mon.a.smithi044.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-17T07:19:43.851 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2024-09-17T07:19:43.851 INFO:journalctl@ceph.mon.a.smithi044.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T07:19:43.851 INFO:journalctl@ceph.mon.a.smithi044.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-09-17T07:19:43.852 INFO:journalctl@ceph.mon.a.smithi044.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T07:19:43.852 INFO:journalctl@ceph.mon.a.smithi044.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T07:19:43.852 INFO:journalctl@ceph.mon.a.smithi044.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T07:19:43.852 INFO:journalctl@ceph.mon.a.smithi044.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-17T07:19:43.852 INFO:journalctl@ceph.mon.a.smithi044.stdout: Cumulative compaction: 0.00 GB write, 165.48 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T07:19:43.852 INFO:journalctl@ceph.mon.a.smithi044.stdout: Interval compaction: 0.00 GB write, 165.48 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T07:19:43.852 INFO:journalctl@ceph.mon.a.smithi044.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T07:19:43.852 INFO:journalctl@ceph.mon.a.smithi044.stdout: Block cache BinnedLRUCache@0x5564232cd350#2 capacity: 512.00 MB usage: 1.70 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-17T07:19:43.852 INFO:journalctl@ceph.mon.a.smithi044.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.38 KB,7.15256e-05%) IndexBlock(1,1.33 KB,0.00025332%) Misc(1,0.00 KB,0%) 2024-09-17T07:19:43.852 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2024-09-17T07:19:43.852 INFO:journalctl@ceph.mon.a.smithi044.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-17T07:19:43.852 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: starting mon.a rank 0 at public addrs [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] at bind addrs [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 817ae996-74c3-11ef-bceb-c7b262605968 2024-09-17T07:19:43.852 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: mon.a@-1(???) e3 preinit fsid 817ae996-74c3-11ef-bceb-c7b262605968 2024-09-17T07:19:43.852 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: mon.a@-1(???).mds e1 new map 2024-09-17T07:19:43.853 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: mon.a@-1(???).mds e1 print_map 2024-09-17T07:19:43.853 INFO:journalctl@ceph.mon.a.smithi044.stdout: e1 2024-09-17T07:19:43.853 INFO:journalctl@ceph.mon.a.smithi044.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-17T07:19:43.853 INFO:journalctl@ceph.mon.a.smithi044.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-17T07:19:43.853 INFO:journalctl@ceph.mon.a.smithi044.stdout: legacy client fscid: -1 2024-09-17T07:19:43.853 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2024-09-17T07:19:43.853 INFO:journalctl@ceph.mon.a.smithi044.stdout: No filesystems configured 2024-09-17T07:19:43.853 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: mon.a@-1(???).osd e75 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-17T07:19:43.853 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: mon.a@-1(???).osd e75 crush map has features 432629239337189376, adjusting msgr requires 2024-09-17T07:19:43.853 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: mon.a@-1(???).osd e75 crush map has features 432629239337189376, adjusting msgr requires 2024-09-17T07:19:43.853 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: mon.a@-1(???).osd e75 crush map has features 432629239337189376, adjusting msgr requires 2024-09-17T07:19:43.853 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 ceph-mon[58191]: mon.a@-1(???).paxosservice(auth 1..23) refresh upgraded, format 0 -> 3 2024-09-17T07:19:43.853 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 bash[58172]: c3293e8cdbbafc442e96bf50af8c9ceec22396bae67d61f59e1f5bae54ba9f2e 2024-09-17T07:19:43.854 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:19:43 smithi044 systemd[1]: Started Ceph mon.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:19:45.183 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: ignoring --setuser ceph since I am not root 2024-09-17T07:19:45.183 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: ignoring --setgroup ceph since I am not root 2024-09-17T07:19:45.183 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:44 smithi179 ceph-mgr[46886]: -- 172.21.15.179:0/3574221885 <== mon.2 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a870dc2d00 con 0x55a870d97000 2024-09-17T07:19:45.183 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:44.977+0000 7f90bfb0c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T07:19:45.183 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:45.033+0000 7f90bfb0c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T07:19:45.183 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:45.181+0000 7f90bfb0c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T07:19:45.183 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:42.277835+0000 mgr.y (mgr.25096) 47 : cluster [DBG] pgmap v24: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2024-09-17T07:19:45.184 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:44.278694+0000 mgr.y (mgr.25096) 48 : cluster [DBG] pgmap v25: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:19:45.184 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:44.817843+0000 mon.a (mon.0) 1 : cluster [INF] mon.a calling monitor election 2024-09-17T07:19:45.184 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:44.819528+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T07:19:45.184 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:44.823479+0000 mon.a (mon.0) 3 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0],b=[v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0],c=[v2:172.21.15.44:3301/0,v1:172.21.15.44:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T07:19:45.184 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:44.824014+0000 mon.a (mon.0) 4 : cluster [DBG] fsmap 2024-09-17T07:19:45.184 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:44.824036+0000 mon.a (mon.0) 5 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-09-17T07:19:45.184 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:44.825149+0000 mon.a (mon.0) 6 : cluster [DBG] mgrmap e24: y(active, since 43s), standbys: x 2024-09-17T07:19:45.184 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:44.825556+0000 mon.a (mon.0) 7 : cluster [WRN] Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-09-17T07:19:45.184 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:44.825569+0000 mon.a (mon.0) 8 : cluster [WRN] [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T07:19:45.184 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:44.825575+0000 mon.a (mon.0) 9 : cluster [WRN] daemon iscsi.foo.smithi044.elmtfk on smithi044 is in unknown state 2024-09-17T07:19:45.184 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:44.825586+0000 mon.a (mon.0) 10 : cluster [WRN] daemon prometheus.a on smithi179 is in unknown state 2024-09-17T07:19:45.184 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:44.834303+0000 mon.a (mon.0) 11 : cluster [DBG] mgrmap e25: y(active, since 43s), standbys: x 2024-09-17T07:19:45.185 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:44.834393+0000 mon.a (mon.0) 12 : audit [INF] from='mgr.25096 ' entity='' 2024-09-17T07:19:45.204 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: ignoring --setuser ceph since I am not root 2024-09-17T07:19:45.204 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: ignoring --setgroup ceph since I am not root 2024-09-17T07:19:45.205 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:44 smithi044 ceph-mgr[53336]: -- 172.21.15.44:0/1935310669 <== mon.0 v2:172.21.15.44:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557063042d00 con 0x557063017000 2024-09-17T07:19:45.205 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:19:44.990+0000 7f5adb778040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T07:19:45.205 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:19:45.054+0000 7f5adb778040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T07:19:45.205 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:42.277835+0000 mgr.y (mgr.25096) 47 : cluster [DBG] pgmap v24: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2024-09-17T07:19:45.205 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:44.278694+0000 mgr.y (mgr.25096) 48 : cluster [DBG] pgmap v25: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:19:45.205 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:44.817843+0000 mon.a (mon.0) 1 : cluster [INF] mon.a calling monitor election 2024-09-17T07:19:45.205 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:44.819528+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T07:19:45.205 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:44.823479+0000 mon.a (mon.0) 3 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0],b=[v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0],c=[v2:172.21.15.44:3301/0,v1:172.21.15.44:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T07:19:45.205 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:44.824014+0000 mon.a (mon.0) 4 : cluster [DBG] fsmap 2024-09-17T07:19:45.205 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:44.824036+0000 mon.a (mon.0) 5 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-09-17T07:19:45.205 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:44.825149+0000 mon.a (mon.0) 6 : cluster [DBG] mgrmap e24: y(active, since 43s), standbys: x 2024-09-17T07:19:45.206 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:44.825556+0000 mon.a (mon.0) 7 : cluster [WRN] Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-09-17T07:19:45.206 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:44.825569+0000 mon.a (mon.0) 8 : cluster [WRN] [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T07:19:45.206 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:44.825575+0000 mon.a (mon.0) 9 : cluster [WRN] daemon iscsi.foo.smithi044.elmtfk on smithi044 is in unknown state 2024-09-17T07:19:45.206 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:44.825586+0000 mon.a (mon.0) 10 : cluster [WRN] daemon prometheus.a on smithi179 is in unknown state 2024-09-17T07:19:45.206 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:44.834303+0000 mon.a (mon.0) 11 : cluster [DBG] mgrmap e25: y(active, since 43s), standbys: x 2024-09-17T07:19:45.206 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:44.834393+0000 mon.a (mon.0) 12 : audit [INF] from='mgr.25096 ' entity='' 2024-09-17T07:19:45.462 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:19:45.225+0000 7f5adb778040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T07:19:45.462 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:19:45.297+0000 7f5adb778040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T07:19:45.528 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:45.247+0000 7f90bfb0c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T07:19:45.528 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:45.389+0000 7f90bfb0c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T07:19:45.772 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:19:45.474+0000 7f5adb778040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T07:19:45.772 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:19:45.622+0000 7f5adb778040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T07:19:45.795 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:45.527+0000 7f90bfb0c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T07:19:45.796 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:45 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Scheduled restart job, restart counter is at 2. 2024-09-17T07:19:45.796 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:45 smithi179 systemd[1]: Stopped Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:19:45.796 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:45 smithi179 systemd[1]: Starting Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:19:46.030 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:19:45.964+0000 7f5adb778040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T07:19:46.030 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:19:46.029+0000 7f5adb778040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T07:19:46.077 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:45.855+0000 7f90bfb0c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T07:19:46.077 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:45 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:45.915+0000 7f90bfb0c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T07:19:46.078 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:45 smithi179 podman[48165]: 2024-09-17 07:19:45.701892594 +0000 UTC m=+0.019997837 image pull de242295e2257c37c8cadfd962369228f8f10b2d48a44259b65fef44ad4f6490 docker.io/prom/prometheus:v2.18.1 2024-09-17T07:19:46.078 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:45 smithi179 podman[48165]: 2024-09-17 07:19:45.825791844 +0000 UTC m=+0.143897064 container create 15d23034ffe1fce08749a390e8826b3f7808284137b0c824531c248cc22f854d (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:19:46.324 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:19:46.229+0000 7f5adb778040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T07:19:46.331 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:46.106+0000 7f90bfb0c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T07:19:46.332 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 podman[48165]: 2024-09-17 07:19:46.078026492 +0000 UTC m=+0.396131724 container init 15d23034ffe1fce08749a390e8826b3f7808284137b0c824531c248cc22f854d (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:19:46.332 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 podman[48165]: 2024-09-17 07:19:46.081232531 +0000 UTC m=+0.399337761 container start 15d23034ffe1fce08749a390e8826b3f7808284137b0c824531c248cc22f854d (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:19:46.332 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[48181]: level=info ts=2024-09-17T07:19:46.112Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-17T07:19:46.332 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[48181]: level=info ts=2024-09-17T07:19:46.112Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2024-09-17T07:19:46.332 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[48181]: level=info ts=2024-09-17T07:19:46.112Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2024-09-17T07:19:46.332 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[48181]: level=info ts=2024-09-17T07:19:46.112Z caller=main.go:339 host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi179 (none))" 2024-09-17T07:19:46.332 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[48181]: level=info ts=2024-09-17T07:19:46.112Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T07:19:46.332 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[48181]: level=info ts=2024-09-17T07:19:46.112Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T07:19:46.332 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[48181]: level=info ts=2024-09-17T07:19:46.114Z caller=main.go:678 msg="Starting TSDB ..." 2024-09-17T07:19:46.333 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[48181]: level=info ts=2024-09-17T07:19:46.114Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2024-09-17T07:19:46.333 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[48181]: level=info ts=2024-09-17T07:19:46.117Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2024-09-17T07:19:46.333 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[48181]: level=info ts=2024-09-17T07:19:46.118Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=9 2024-09-17T07:19:46.333 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[48181]: level=info ts=2024-09-17T07:19:46.127Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=9 2024-09-17T07:19:46.333 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[48181]: level=info ts=2024-09-17T07:19:46.127Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=9 2024-09-17T07:19:46.333 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[48181]: level=info ts=2024-09-17T07:19:46.128Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=9 2024-09-17T07:19:46.333 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[48181]: level=info ts=2024-09-17T07:19:46.128Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=9 2024-09-17T07:19:46.333 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[48181]: level=info ts=2024-09-17T07:19:46.128Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=9 2024-09-17T07:19:46.334 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[48181]: level=info ts=2024-09-17T07:19:46.128Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=9 2024-09-17T07:19:46.334 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[48181]: level=info ts=2024-09-17T07:19:46.128Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=7 maxSegment=9 2024-09-17T07:19:46.334 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[48181]: level=info ts=2024-09-17T07:19:46.128Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=8 maxSegment=9 2024-09-17T07:19:46.334 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[48181]: level=info ts=2024-09-17T07:19:46.129Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=9 maxSegment=9 2024-09-17T07:19:46.334 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[48181]: level=info ts=2024-09-17T07:19:46.129Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=11.328334ms 2024-09-17T07:19:46.334 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[48181]: level=info ts=2024-09-17T07:19:46.130Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2024-09-17T07:19:46.334 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[48181]: level=info ts=2024-09-17T07:19:46.130Z caller=main.go:695 msg="TSDB started" 2024-09-17T07:19:46.334 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[48181]: level=info ts=2024-09-17T07:19:46.130Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T07:19:46.334 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[48181]: level=info ts=2024-09-17T07:19:46.131Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2024-09-17T07:19:46.334 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[48181]: level=info ts=2024-09-17T07:19:46.131Z caller=main.go:561 msg="Stopping notify discovery manager..." 2024-09-17T07:19:46.334 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[48181]: level=info ts=2024-09-17T07:19:46.131Z caller=main.go:583 msg="Stopping scrape manager..." 2024-09-17T07:19:46.335 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[48181]: level=info ts=2024-09-17T07:19:46.131Z caller=main.go:557 msg="Notify discovery manager stopped" 2024-09-17T07:19:46.335 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[48181]: level=info ts=2024-09-17T07:19:46.131Z caller=main.go:543 msg="Scrape discovery manager stopped" 2024-09-17T07:19:46.335 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[48181]: level=info ts=2024-09-17T07:19:46.131Z caller=main.go:577 msg="Scrape manager stopped" 2024-09-17T07:19:46.335 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[48181]: level=info ts=2024-09-17T07:19:46.131Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2024-09-17T07:19:46.335 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[48181]: level=info ts=2024-09-17T07:19:46.131Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2024-09-17T07:19:46.335 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[48181]: level=info ts=2024-09-17T07:19:46.131Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2024-09-17T07:19:46.335 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[48181]: level=info ts=2024-09-17T07:19:46.131Z caller=main.go:749 msg="Notifier manager stopped" 2024-09-17T07:19:46.335 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[48181]: level=error ts=2024-09-17T07:19:46.131Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 14: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain\n line 31: field http_sd_configs not found in type config.plain\n line 45: field http_sd_configs not found in type config.plain\n line 49: field http_sd_configs not found in type config.plain" 2024-09-17T07:19:46.335 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 bash[48165]: 15d23034ffe1fce08749a390e8826b3f7808284137b0c824531c248cc22f854d 2024-09-17T07:19:46.335 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 systemd[1]: Started Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:19:46.335 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 podman[48199]: 2024-09-17 07:19:46.210346695 +0000 UTC m=+0.071036439 container died 15d23034ffe1fce08749a390e8826b3f7808284137b0c824531c248cc22f854d (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:19:47.109 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:47.040+0000 7f90bfb0c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T07:19:47.109 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:45.840324+0000 mon.a (mon.0) 13 : cluster [DBG] mgrmap e26: y(active, since 44s), standbys: x 2024-09-17T07:19:47.109 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:46 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:46.546657+0000 mon.c (mon.1) 89 : audit [DBG] from='client.? 172.21.15.44:0/850084046' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:19:47.110 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 podman[48199]: 2024-09-17 07:19:46.857858403 +0000 UTC m=+0.718548136 container remove 15d23034ffe1fce08749a390e8826b3f7808284137b0c824531c248cc22f854d (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:19:47.110 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2024-09-17T07:19:47.110 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:46 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Failed with result 'exit-code'. 2024-09-17T07:19:47.171 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:45.840324+0000 mon.a (mon.0) 13 : cluster [DBG] mgrmap e26: y(active, since 44s), standbys: x 2024-09-17T07:19:47.171 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:46.546657+0000 mon.c (mon.1) 89 : audit [DBG] from='client.? 172.21.15.44:0/850084046' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:19:47.364 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:47.108+0000 7f90bfb0c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T07:19:47.574 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:19:47.170+0000 7f5adb778040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T07:19:47.574 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:19:47.242+0000 7f5adb778040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T07:19:48.000 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:46.892146+0000 mon.a (mon.0) 14 : audit [INF] from='client.44136 172.21.15.44:0/2861191450' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:6800/2448217567"}]: access denied 2024-09-17T07:19:48.077 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:46.892146+0000 mon.a (mon.0) 14 : audit [INF] from='client.44136 172.21.15.44:0/2861191450' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:6800/2448217567"}]: access denied 2024-09-17T07:19:48.077 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:47.818+0000 7f90bfb0c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T07:19:48.077 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:47.935+0000 7f90bfb0c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T07:19:48.255 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:19:47.999+0000 7f5adb778040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T07:19:48.256 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:19:48.120+0000 7f5adb778040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T07:19:48.350 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:48.076+0000 7f90bfb0c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T07:19:48.350 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:48.137+0000 7f90bfb0c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T07:19:48.350 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:48.283+0000 7f90bfb0c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T07:19:48.350 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:48.349+0000 7f90bfb0c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T07:19:48.562 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:19:48.272+0000 7f5adb778040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T07:19:48.563 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:19:48.336+0000 7f5adb778040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T07:19:48.563 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:19:48.491+0000 7f5adb778040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T07:19:48.563 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:19:48.561+0000 7f5adb778040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T07:19:48.614 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:48.408+0000 7f90bfb0c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T07:19:48.614 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:48.563+0000 7f90bfb0c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T07:19:48.824 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:19:48.624+0000 7f5adb778040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T07:19:48.824 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:19:48.787+0000 7f5adb778040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T07:19:48.895 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:48.628+0000 7f90bfb0c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T07:19:48.895 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:48.693+0000 7f90bfb0c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T07:19:48.896 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:48.774+0000 7f90bfb0c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T07:19:48.896 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:48.834+0000 7f90bfb0c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T07:19:48.896 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:48.894+0000 7f90bfb0c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T07:19:49.136 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:19:48.855+0000 7f5adb778040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T07:19:49.136 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:19:48.926+0000 7f5adb778040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T07:19:49.136 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:19:49.012+0000 7f5adb778040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T07:19:49.136 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:19:49.074+0000 7f5adb778040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T07:19:49.136 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:19:49.135+0000 7f5adb778040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T07:19:50.074 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:19:49.953+0000 7f5adb778040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T07:19:50.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:49.664148+0000 mon.a (mon.0) 15 : cluster [DBG] Standby manager daemon x restarted 2024-09-17T07:19:50.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:49.664234+0000 mon.a (mon.0) 16 : cluster [DBG] Standby manager daemon x started 2024-09-17T07:19:50.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:49.665106+0000 mon.b (mon.2) 88 : audit [DBG] from='mgr.? 172.21.15.179:0/2076877447' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T07:19:50.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:49.665843+0000 mon.b (mon.2) 89 : audit [DBG] from='mgr.? 172.21.15.179:0/2076877447' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T07:19:50.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:49.666462+0000 mon.b (mon.2) 90 : audit [DBG] from='mgr.? 172.21.15.179:0/2076877447' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T07:19:50.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:49.666702+0000 mon.b (mon.2) 91 : audit [DBG] from='mgr.? 172.21.15.179:0/2076877447' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T07:19:50.114 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:19:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x[46882]: 2024-09-17T07:19:49.661+0000 7f90bfb0c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T07:19:50.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:49.664148+0000 mon.a (mon.0) 15 : cluster [DBG] Standby manager daemon x restarted 2024-09-17T07:19:50.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:49.664234+0000 mon.a (mon.0) 16 : cluster [DBG] Standby manager daemon x started 2024-09-17T07:19:50.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:49.665106+0000 mon.b (mon.2) 88 : audit [DBG] from='mgr.? 172.21.15.179:0/2076877447' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T07:19:50.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:49.665843+0000 mon.b (mon.2) 89 : audit [DBG] from='mgr.? 172.21.15.179:0/2076877447' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T07:19:50.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:49.666462+0000 mon.b (mon.2) 90 : audit [DBG] from='mgr.? 172.21.15.179:0/2076877447' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T07:19:50.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:49.666702+0000 mon.b (mon.2) 91 : audit [DBG] from='mgr.? 172.21.15.179:0/2076877447' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T07:19:50.989 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:19:50.972+0000 7f5a7ae47640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-17T07:19:50.989 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-09-17T07:19:50.991 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:49.719390+0000 mon.a (mon.0) 17 : cluster [DBG] mgrmap e27: y(active, since 47s), standbys: x 2024-09-17T07:19:50.991 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:49.955368+0000 mon.a (mon.0) 18 : cluster [INF] Active manager daemon y restarted 2024-09-17T07:19:50.991 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:49.955808+0000 mon.a (mon.0) 19 : cluster [INF] Activating manager daemon y 2024-09-17T07:19:50.991 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:49.966186+0000 mon.a (mon.0) 20 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-09-17T07:19:50.991 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:49.968502+0000 mon.a (mon.0) 21 : cluster [DBG] mgrmap e28: y(active, starting, since 0.0127954s), standbys: x 2024-09-17T07:19:50.991 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:49.971627+0000 mon.a (mon.0) 22 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:19:50.991 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:49.971811+0000 mon.a (mon.0) 23 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:19:50.991 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:49.971998+0000 mon.a (mon.0) 24 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:19:50.991 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:49.972996+0000 mon.a (mon.0) 25 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T07:19:50.991 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:49.973149+0000 mon.a (mon.0) 26 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T07:19:50.991 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:49.973318+0000 mon.a (mon.0) 27 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T07:19:50.991 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:49.973506+0000 mon.a (mon.0) 28 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T07:19:50.992 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:49.973688+0000 mon.a (mon.0) 29 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:19:50.992 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:49.973908+0000 mon.a (mon.0) 30 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:19:50.992 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:49.974064+0000 mon.a (mon.0) 31 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:19:50.992 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:49.974257+0000 mon.a (mon.0) 32 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:19:50.992 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:49.974434+0000 mon.a (mon.0) 33 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:19:50.992 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:49.974607+0000 mon.a (mon.0) 34 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:19:50.992 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:49.974800+0000 mon.a (mon.0) 35 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T07:19:50.992 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:49.974936+0000 mon.a (mon.0) 36 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T07:19:50.992 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:49.975331+0000 mon.a (mon.0) 37 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T07:19:50.992 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:49.982013+0000 mon.a (mon.0) 38 : cluster [INF] Manager daemon y is now available 2024-09-17T07:19:50.992 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:49.999994+0000 mon.a (mon.0) 39 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:19:50.992 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:50.024202+0000 mon.a (mon.0) 40 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:19:50.993 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:50.045867+0000 mon.a (mon.0) 41 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:19:51.068 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:49.719390+0000 mon.a (mon.0) 17 : cluster [DBG] mgrmap e27: y(active, since 47s), standbys: x 2024-09-17T07:19:51.069 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:49.955368+0000 mon.a (mon.0) 18 : cluster [INF] Active manager daemon y restarted 2024-09-17T07:19:51.069 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:49.955808+0000 mon.a (mon.0) 19 : cluster [INF] Activating manager daemon y 2024-09-17T07:19:51.069 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:49.966186+0000 mon.a (mon.0) 20 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-09-17T07:19:51.069 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:49.968502+0000 mon.a (mon.0) 21 : cluster [DBG] mgrmap e28: y(active, starting, since 0.0127954s), standbys: x 2024-09-17T07:19:51.069 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:49.971627+0000 mon.a (mon.0) 22 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:19:51.069 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:49.971811+0000 mon.a (mon.0) 23 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:19:51.069 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:49.971998+0000 mon.a (mon.0) 24 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:19:51.069 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:49.972996+0000 mon.a (mon.0) 25 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T07:19:51.069 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:49.973149+0000 mon.a (mon.0) 26 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T07:19:51.069 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:49.973318+0000 mon.a (mon.0) 27 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T07:19:51.069 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:49.973506+0000 mon.a (mon.0) 28 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T07:19:51.069 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:49.973688+0000 mon.a (mon.0) 29 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:19:51.070 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:49.973908+0000 mon.a (mon.0) 30 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:19:51.070 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:49.974064+0000 mon.a (mon.0) 31 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:19:51.070 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:49.974257+0000 mon.a (mon.0) 32 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:19:51.070 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:49.974434+0000 mon.a (mon.0) 33 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:19:51.070 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:49.974607+0000 mon.a (mon.0) 34 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:19:51.070 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:49.974800+0000 mon.a (mon.0) 35 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T07:19:51.070 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:49.974936+0000 mon.a (mon.0) 36 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T07:19:51.070 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:49.975331+0000 mon.a (mon.0) 37 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T07:19:51.070 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:49.982013+0000 mon.a (mon.0) 38 : cluster [INF] Manager daemon y is now available 2024-09-17T07:19:51.070 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:49.999994+0000 mon.a (mon.0) 39 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:19:51.070 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:50.024202+0000 mon.a (mon.0) 40 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:19:51.070 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:50 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:50.045867+0000 mon.a (mon.0) 41 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:19:51.698 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T07:19:51.698 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.a smithi044 *:9093,9094 running (96s) 15s ago 6m 13.0M - 0.20.0 0881eb8f169f 0473d4b023ad 2024-09-17T07:19:51.698 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.a smithi179 *:3000 running (67s) 17s ago 6m 26.7M - 6.7.4 557c83e11646 43ab0e03dcac 2024-09-17T07:19:51.698 INFO:teuthology.orchestra.run.smithi044.stdout:iscsi.foo.smithi044.elmtfk smithi044 unknown 15s ago 5m - - 2024-09-17T07:19:51.698 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.x smithi179 *:8443,8765 running (23s) 17s ago 10m 66.2M - 18.2.4-855-g19e2a2be 980a0bfcd684 66b52dc26022 2024-09-17T07:19:51.698 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.y smithi044 *:8443,8765 running (60s) 15s ago 11m 476M - 18.2.4-855-g19e2a2be 980a0bfcd684 8080dd318a1c 2024-09-17T07:19:51.699 INFO:teuthology.orchestra.run.smithi044.stdout:mon.a smithi044 running (11m) 15s ago 11m 61.3M 2048M 16.2.0 24ecd6d5f14c 7905efa25a24 2024-09-17T07:19:51.699 INFO:teuthology.orchestra.run.smithi044.stdout:mon.b smithi179 running (10m) 17s ago 10m 60.7M 2048M 16.2.0 24ecd6d5f14c 198f81fe2641 2024-09-17T07:19:51.699 INFO:teuthology.orchestra.run.smithi044.stdout:mon.c smithi044 running (10m) 15s ago 10m 40.7M 2048M 16.2.0 24ecd6d5f14c e0d29cfe87a2 2024-09-17T07:19:51.699 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.a smithi044 *:9100 running (105s) 15s ago 6m 17.0M - 0.18.1 e5a616e4b9cf 8a2cde5bd9ed 2024-09-17T07:19:51.699 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.b smithi179 *:9100 running (79s) 17s ago 7m 4710k - 0.18.1 e5a616e4b9cf 04c63b5cb524 2024-09-17T07:19:51.699 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (10m) 15s ago 10m 47.9M 4096M 16.2.0 24ecd6d5f14c 4517de6029ac 2024-09-17T07:19:51.699 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (9m) 15s ago 9m 45.5M 4096M 16.2.0 24ecd6d5f14c 2d5122cbeff9 2024-09-17T07:19:51.699 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (9m) 15s ago 9m 43.1M 4096M 16.2.0 24ecd6d5f14c 7fd2f30be874 2024-09-17T07:19:51.699 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi044 running (9m) 15s ago 9m 48.3M 4096M 16.2.0 24ecd6d5f14c 2b7d623395a7 2024-09-17T07:19:51.699 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi179 running (8m) 17s ago 8m 51.3M 4096M 16.2.0 24ecd6d5f14c 911e6f47c822 2024-09-17T07:19:51.699 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi179 running (8m) 17s ago 8m 47.5M 4096M 16.2.0 24ecd6d5f14c e56060742dff 2024-09-17T07:19:51.699 INFO:teuthology.orchestra.run.smithi044.stdout:osd.6 smithi179 running (8m) 17s ago 8m 47.0M 4096M 16.2.0 24ecd6d5f14c 7a5c30b68cbc 2024-09-17T07:19:51.699 INFO:teuthology.orchestra.run.smithi044.stdout:osd.7 smithi179 running (7m) 17s ago 7m 53.0M 4096M 16.2.0 24ecd6d5f14c b0d186fbf8a6 2024-09-17T07:19:51.699 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.a smithi179 *:9095 unknown 17s ago 7m - - 2024-09-17T07:19:51.700 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi044.nhyhnh smithi044 *:8000 running (5m) 15s ago 5m 37.8M - 16.2.0 24ecd6d5f14c 9b455479614f 2024-09-17T07:19:51.700 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi179.tqlzyy smithi179 *:8000 running (5m) 17s ago 5m 40.5M - 16.2.0 24ecd6d5f14c c8d1693241ce 2024-09-17T07:19:51.700 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi044.vnufiz smithi044 *:80 running (5m) 15s ago 5m 37.6M - 16.2.0 24ecd6d5f14c 6eae2bff5ec8 2024-09-17T07:19:51.700 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi179.ecjhll smithi179 *:80 running (5m) 17s ago 5m 41.0M - 16.2.0 24ecd6d5f14c 253109ee0394 2024-09-17T07:19:51.973 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:50.967342+0000 mon.a (mon.0) 42 : cluster [DBG] mgrmap e29: y(active, since 1.01164s), standbys: x 2024-09-17T07:19:51.973 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:19:51.783085+0000 mgr.y (mgr.44106) 5 : cephadm [INF] [17/Sep/2024:07:19:51] ENGINE Bus STARTING 2024-09-17T07:19:51.974 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:19:51.905+0000 7f5a41542640 -1 log_channel(cephadm) log [ERR] : [17/Sep/2024:07:19:51] ENGINE Error in HTTPServer.serve 2024-09-17T07:19:51.975 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: Traceback (most recent call last): 2024-09-17T07:19:51.975 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T07:19:51.975 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: self._connections.run(self.expiration_interval) 2024-09-17T07:19:51.975 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T07:19:51.975 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: self._run(expiration_interval) 2024-09-17T07:19:51.975 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T07:19:51.975 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T07:19:51.975 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T07:19:51.975 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T07:19:51.975 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T07:19:51.975 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: s = self.context.wrap_socket( 2024-09-17T07:19:51.975 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T07:19:51.975 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: return self.sslsocket_class._create( 2024-09-17T07:19:51.976 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T07:19:51.976 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: self.do_handshake() 2024-09-17T07:19:51.976 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T07:19:51.976 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: self._sslobj.do_handshake() 2024-09-17T07:19:51.976 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T07:19:51.976 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:19:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:19:52.120 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:19:52.120 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-09-17T07:19:52.121 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 2, 2024-09-17T07:19:52.121 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 1 2024-09-17T07:19:52.121 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:19:52.121 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-09-17T07:19:52.121 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T07:19:52.121 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:19:52.121 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-09-17T07:19:52.121 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2024-09-17T07:19:52.121 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:19:52.121 INFO:teuthology.orchestra.run.smithi044.stdout: "mds": {}, 2024-09-17T07:19:52.121 INFO:teuthology.orchestra.run.smithi044.stdout: "rgw": { 2024-09-17T07:19:52.121 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2024-09-17T07:19:52.121 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:19:52.121 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-09-17T07:19:52.121 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 14, 2024-09-17T07:19:52.122 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3 2024-09-17T07:19:52.122 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-09-17T07:19:52.122 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:19:52.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:50.967342+0000 mon.a (mon.0) 42 : cluster [DBG] mgrmap e29: y(active, since 1.01164s), standbys: x 2024-09-17T07:19:52.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:51 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:19:51.783085+0000 mgr.y (mgr.44106) 5 : cephadm [INF] [17/Sep/2024:07:19:51] ENGINE Bus STARTING 2024-09-17T07:19:52.484 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:19:52.485 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9", 2024-09-17T07:19:52.485 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-09-17T07:19:52.485 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T07:19:52.485 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [ 2024-09-17T07:19:52.485 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr" 2024-09-17T07:19:52.485 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-09-17T07:19:52.485 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "2/23 daemons upgraded", 2024-09-17T07:19:52.485 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "", 2024-09-17T07:19:52.485 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-09-17T07:19:52.485 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:19:53.002 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_OK 2024-09-17T07:19:53.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:19:51.902932+0000 mgr.y (mgr.44106) 6 : cephadm [INF] [17/Sep/2024:07:19:51] ENGINE Serving on https://172.21.15.44:7150 2024-09-17T07:19:53.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:19:51.906219+0000 mgr.y (mgr.44106) 7 : cephadm [ERR] [17/Sep/2024:07:19:51] ENGINE Error in HTTPServer.serve 2024-09-17T07:19:53.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: Traceback (most recent call last): 2024-09-17T07:19:53.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T07:19:53.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: self._connections.run(self.expiration_interval) 2024-09-17T07:19:53.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T07:19:53.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: self._run(expiration_interval) 2024-09-17T07:19:53.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T07:19:53.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T07:19:53.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T07:19:53.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T07:19:53.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T07:19:53.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: s = self.context.wrap_socket( 2024-09-17T07:19:53.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T07:19:53.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: return self.sslsocket_class._create( 2024-09-17T07:19:53.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T07:19:53.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: self.do_handshake() 2024-09-17T07:19:53.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T07:19:53.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: self._sslobj.do_handshake() 2024-09-17T07:19:53.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T07:19:53.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: 2024-09-17T07:19:53.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:51.962823+0000 mon.a (mon.0) 43 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2024-09-17T07:19:53.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:51.962845+0000 mon.a (mon.0) 44 : cluster [INF] Cluster is now healthy 2024-09-17T07:19:53.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:51.974381+0000 mgr.y (mgr.44106) 8 : cluster [DBG] pgmap v4: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:19:53.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:51.977292+0000 mon.a (mon.0) 45 : cluster [DBG] mgrmap e30: y(active, since 2s), standbys: x 2024-09-17T07:19:53.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:19:52.004430+0000 mgr.y (mgr.44106) 9 : cephadm [INF] [17/Sep/2024:07:19:52] ENGINE Serving on http://172.21.15.44:8765 2024-09-17T07:19:53.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:19:52.004553+0000 mgr.y (mgr.44106) 10 : cephadm [INF] [17/Sep/2024:07:19:52] ENGINE Bus STARTED 2024-09-17T07:19:53.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:52.120545+0000 mon.c (mon.1) 90 : audit [DBG] from='client.? 172.21.15.44:0/2556707789' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:19:53.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:52.484689+0000 mgr.y (mgr.44106) 11 : audit [DBG] from='client.25351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:19:53.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:19:51.902932+0000 mgr.y (mgr.44106) 6 : cephadm [INF] [17/Sep/2024:07:19:51] ENGINE Serving on https://172.21.15.44:7150 2024-09-17T07:19:53.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:19:51.906219+0000 mgr.y (mgr.44106) 7 : cephadm [ERR] [17/Sep/2024:07:19:51] ENGINE Error in HTTPServer.serve 2024-09-17T07:19:53.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: Traceback (most recent call last): 2024-09-17T07:19:53.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T07:19:53.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: self._connections.run(self.expiration_interval) 2024-09-17T07:19:53.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T07:19:53.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: self._run(expiration_interval) 2024-09-17T07:19:53.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T07:19:53.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T07:19:53.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T07:19:53.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T07:19:53.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T07:19:53.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: s = self.context.wrap_socket( 2024-09-17T07:19:53.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T07:19:53.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: return self.sslsocket_class._create( 2024-09-17T07:19:53.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T07:19:53.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: self.do_handshake() 2024-09-17T07:19:53.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T07:19:53.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: self._sslobj.do_handshake() 2024-09-17T07:19:53.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T07:19:53.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: 2024-09-17T07:19:53.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:51.962823+0000 mon.a (mon.0) 43 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2024-09-17T07:19:53.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:51.962845+0000 mon.a (mon.0) 44 : cluster [INF] Cluster is now healthy 2024-09-17T07:19:53.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:51.974381+0000 mgr.y (mgr.44106) 8 : cluster [DBG] pgmap v4: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:19:53.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:51.977292+0000 mon.a (mon.0) 45 : cluster [DBG] mgrmap e30: y(active, since 2s), standbys: x 2024-09-17T07:19:53.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:19:52.004430+0000 mgr.y (mgr.44106) 9 : cephadm [INF] [17/Sep/2024:07:19:52] ENGINE Serving on http://172.21.15.44:8765 2024-09-17T07:19:53.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:19:52.004553+0000 mgr.y (mgr.44106) 10 : cephadm [INF] [17/Sep/2024:07:19:52] ENGINE Bus STARTED 2024-09-17T07:19:53.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:52.120545+0000 mon.c (mon.1) 90 : audit [DBG] from='client.? 172.21.15.44:0/2556707789' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:19:53.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:52 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:52.484689+0000 mgr.y (mgr.44106) 11 : audit [DBG] from='client.25351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:19:54.314 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:53 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:53.003035+0000 mon.a (mon.0) 46 : audit [DBG] from='client.44190 172.21.15.44:0/2380431437' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:19:54.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:53 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:53.003035+0000 mon.a (mon.0) 46 : audit [DBG] from='client.44190 172.21.15.44:0/2380431437' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:19:55.324 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:53.974789+0000 mgr.y (mgr.44106) 12 : cluster [DBG] pgmap v5: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:19:55.324 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:53.983503+0000 mon.a (mon.0) 47 : cluster [DBG] mgrmap e31: y(active, since 4s), standbys: x 2024-09-17T07:19:55.363 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:53.974789+0000 mgr.y (mgr.44106) 12 : cluster [DBG] pgmap v5: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:19:55.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:53.983503+0000 mon.a (mon.0) 47 : cluster [DBG] mgrmap e31: y(active, since 4s), standbys: x 2024-09-17T07:19:56.324 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:55.124695+0000 mon.a (mon.0) 48 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:19:56.324 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:55.130221+0000 mon.a (mon.0) 49 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:19:56.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:55.328730+0000 mon.a (mon.0) 50 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:19:56.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:55.333503+0000 mon.a (mon.0) 51 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:19:56.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:55.648329+0000 mon.a (mon.0) 52 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:19:56.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:55.654151+0000 mon.a (mon.0) 53 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:19:56.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:55.868079+0000 mon.a (mon.0) 54 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:19:56.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:55.876549+0000 mon.a (mon.0) 55 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:19:56.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:56 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:55.124695+0000 mon.a (mon.0) 48 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:19:56.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:56 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:55.130221+0000 mon.a (mon.0) 49 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:19:56.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:56 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:55.328730+0000 mon.a (mon.0) 50 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:19:56.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:56 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:55.333503+0000 mon.a (mon.0) 51 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:19:56.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:56 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:55.648329+0000 mon.a (mon.0) 52 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:19:56.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:56 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:55.654151+0000 mon.a (mon.0) 53 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:19:56.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:56 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:55.868079+0000 mon.a (mon.0) 54 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:19:56.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:56 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:55.876549+0000 mon.a (mon.0) 55 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:19:57.063 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Scheduled restart job, restart counter is at 3. 2024-09-17T07:19:57.063 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 systemd[1]: Stopped Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:19:57.063 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 systemd[1]: Starting Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:19:57.364 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 podman[49248]: 2024-09-17 07:19:57.119196653 +0000 UTC m=+0.022601184 image pull de242295e2257c37c8cadfd962369228f8f10b2d48a44259b65fef44ad4f6490 docker.io/prom/prometheus:v2.18.1 2024-09-17T07:19:57.364 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 podman[49248]: 2024-09-17 07:19:57.239154404 +0000 UTC m=+0.142558954 container create 1d46e55c1d14ce0caa579dbd2b7b59f4e486b1480db1ba9a3047943873f3b410 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:19:57.864 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 podman[49248]: 2024-09-17 07:19:57.49909512 +0000 UTC m=+0.402499635 container init 1d46e55c1d14ce0caa579dbd2b7b59f4e486b1480db1ba9a3047943873f3b410 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:19:57.865 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 podman[49248]: 2024-09-17 07:19:57.502107416 +0000 UTC m=+0.405511931 container start 1d46e55c1d14ce0caa579dbd2b7b59f4e486b1480db1ba9a3047943873f3b410 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:19:57.865 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49281]: level=info ts=2024-09-17T07:19:57.532Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-17T07:19:57.865 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49281]: level=info ts=2024-09-17T07:19:57.532Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2024-09-17T07:19:57.865 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49281]: level=info ts=2024-09-17T07:19:57.532Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2024-09-17T07:19:57.865 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49281]: level=info ts=2024-09-17T07:19:57.532Z caller=main.go:339 host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi179 (none))" 2024-09-17T07:19:57.865 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49281]: level=info ts=2024-09-17T07:19:57.532Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T07:19:57.865 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49281]: level=info ts=2024-09-17T07:19:57.532Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T07:19:57.865 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49281]: level=info ts=2024-09-17T07:19:57.533Z caller=main.go:678 msg="Starting TSDB ..." 2024-09-17T07:19:57.865 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49281]: level=info ts=2024-09-17T07:19:57.533Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2024-09-17T07:19:57.865 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49281]: level=info ts=2024-09-17T07:19:57.535Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2024-09-17T07:19:57.865 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49281]: level=info ts=2024-09-17T07:19:57.536Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=10 2024-09-17T07:19:57.866 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49281]: level=info ts=2024-09-17T07:19:57.544Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=10 2024-09-17T07:19:57.866 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49281]: level=info ts=2024-09-17T07:19:57.544Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=10 2024-09-17T07:19:57.866 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49281]: level=info ts=2024-09-17T07:19:57.544Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=10 2024-09-17T07:19:57.866 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49281]: level=info ts=2024-09-17T07:19:57.544Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=10 2024-09-17T07:19:57.866 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49281]: level=info ts=2024-09-17T07:19:57.544Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=10 2024-09-17T07:19:57.866 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49281]: level=info ts=2024-09-17T07:19:57.544Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=10 2024-09-17T07:19:57.866 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49281]: level=info ts=2024-09-17T07:19:57.544Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=7 maxSegment=10 2024-09-17T07:19:57.866 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49281]: level=info ts=2024-09-17T07:19:57.545Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=8 maxSegment=10 2024-09-17T07:19:57.866 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49281]: level=info ts=2024-09-17T07:19:57.545Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=9 maxSegment=10 2024-09-17T07:19:57.866 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49281]: level=info ts=2024-09-17T07:19:57.545Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=10 maxSegment=10 2024-09-17T07:19:57.867 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49281]: level=info ts=2024-09-17T07:19:57.545Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=9.50794ms 2024-09-17T07:19:57.867 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49281]: level=info ts=2024-09-17T07:19:57.548Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2024-09-17T07:19:57.867 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49281]: level=info ts=2024-09-17T07:19:57.548Z caller=main.go:695 msg="TSDB started" 2024-09-17T07:19:57.867 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49281]: level=info ts=2024-09-17T07:19:57.548Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T07:19:57.867 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49281]: level=info ts=2024-09-17T07:19:57.548Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2024-09-17T07:19:57.867 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49281]: level=info ts=2024-09-17T07:19:57.548Z caller=main.go:561 msg="Stopping notify discovery manager..." 2024-09-17T07:19:57.867 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49281]: level=info ts=2024-09-17T07:19:57.548Z caller=main.go:583 msg="Stopping scrape manager..." 2024-09-17T07:19:57.867 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49281]: level=info ts=2024-09-17T07:19:57.548Z caller=main.go:557 msg="Notify discovery manager stopped" 2024-09-17T07:19:57.867 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49281]: level=info ts=2024-09-17T07:19:57.548Z caller=main.go:543 msg="Scrape discovery manager stopped" 2024-09-17T07:19:57.867 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49281]: level=info ts=2024-09-17T07:19:57.548Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2024-09-17T07:19:57.867 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49281]: level=info ts=2024-09-17T07:19:57.548Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2024-09-17T07:19:57.867 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49281]: level=info ts=2024-09-17T07:19:57.548Z caller=main.go:577 msg="Scrape manager stopped" 2024-09-17T07:19:57.867 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49281]: level=info ts=2024-09-17T07:19:57.549Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2024-09-17T07:19:57.867 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49281]: level=info ts=2024-09-17T07:19:57.549Z caller=main.go:749 msg="Notifier manager stopped" 2024-09-17T07:19:57.868 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49281]: level=error ts=2024-09-17T07:19:57.549Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 14: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain\n line 31: field http_sd_configs not found in type config.plain\n line 45: field http_sd_configs not found in type config.plain\n line 49: field http_sd_configs not found in type config.plain" 2024-09-17T07:19:57.868 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 bash[49248]: 1d46e55c1d14ce0caa579dbd2b7b59f4e486b1480db1ba9a3047943873f3b410 2024-09-17T07:19:57.868 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 systemd[1]: Started Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:19:57.868 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:57 smithi179 podman[49298]: 2024-09-17 07:19:57.573358907 +0000 UTC m=+0.016809588 container died 1d46e55c1d14ce0caa579dbd2b7b59f4e486b1480db1ba9a3047943873f3b410 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:19:58.130 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:57 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:55.975272+0000 mgr.y (mgr.44106) 13 : cluster [DBG] pgmap v6: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:19:58.130 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:58 smithi179 podman[49298]: 2024-09-17 07:19:58.111306938 +0000 UTC m=+0.554757665 container remove 1d46e55c1d14ce0caa579dbd2b7b59f4e486b1480db1ba9a3047943873f3b410 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:19:58.131 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:58 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2024-09-17T07:19:58.324 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:57 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:55.975272+0000 mgr.y (mgr.44106) 13 : cluster [DBG] pgmap v6: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:19:58.386 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:19:58 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Failed with result 'exit-code'. 2024-09-17T07:20:00.312 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:19:59 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:57.976025+0000 mgr.y (mgr.44106) 14 : cluster [DBG] pgmap v7: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-17T07:20:00.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:19:59 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:57.976025+0000 mgr.y (mgr.44106) 14 : cluster [DBG] pgmap v7: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-17T07:20:00.883 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:59.923980+0000 mon.c (mon.1) 91 : audit [DBG] from='client.? 172.21.15.44:0/3155806008' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:20:00.884 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:59.971684+0000 mon.a (mon.0) 56 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:00.884 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:59.976918+0000 mon.a (mon.0) 57 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:00.884 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:59.978465+0000 mon.a (mon.0) 58 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:00.884 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:59.979227+0000 mon.a (mon.0) 59 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:20:00.884 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:19:59.984013+0000 mon.a (mon.0) 60 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:00.885 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:20:00.000102+0000 mon.a (mon.0) 61 : cluster [INF] overall HEALTH_OK 2024-09-17T07:20:00.885 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:20:00.124706+0000 mon.a (mon.0) 62 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:20:00.885 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:20:00.126864+0000 mon.a (mon.0) 63 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:20:00.885 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:20:00.127889+0000 mon.a (mon.0) 64 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T07:20:01.309 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:20:00.128510+0000 mon.a (mon.0) 65 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-17T07:20:01.309 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:20:00.306628+0000 mon.c (mon.1) 92 : audit [INF] from='client.? 172.21.15.44:0/1958833622' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2084457509"}]: access denied 2024-09-17T07:20:01.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:00 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:59.923980+0000 mon.c (mon.1) 91 : audit [DBG] from='client.? 172.21.15.44:0/3155806008' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:20:01.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:00 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:59.971684+0000 mon.a (mon.0) 56 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:01.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:00 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:59.976918+0000 mon.a (mon.0) 57 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:01.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:00 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:59.978465+0000 mon.a (mon.0) 58 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:01.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:00 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:59.979227+0000 mon.a (mon.0) 59 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:20:01.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:00 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:19:59.984013+0000 mon.a (mon.0) 60 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:01.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:00 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:20:00.000102+0000 mon.a (mon.0) 61 : cluster [INF] overall HEALTH_OK 2024-09-17T07:20:01.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:00 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:20:00.124706+0000 mon.a (mon.0) 62 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:20:01.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:00 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:20:00.126864+0000 mon.a (mon.0) 63 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:20:01.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:00 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:20:00.127889+0000 mon.a (mon.0) 64 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T07:20:01.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:00 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:20:00.128510+0000 mon.a (mon.0) 65 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-17T07:20:01.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:00 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:20:00.306628+0000 mon.c (mon.1) 92 : audit [INF] from='client.? 172.21.15.44:0/1958833622' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2084457509"}]: access denied 2024-09-17T07:20:02.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:59.976338+0000 mgr.y (mgr.44106) 15 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-17T07:20:02.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:19:59.979966+0000 mgr.y (mgr.44106) 16 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:20:02.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:19:59.980543+0000 mgr.y (mgr.44106) 17 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-17T07:20:02.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cephadm 2024-09-17T07:20:00.128978+0000 mgr.y (mgr.44106) 18 : cephadm [INF] Upgrade: It appears safe to stop mon.c 2024-09-17T07:20:02.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:01 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:20:00.975065+0000 mon.a (mon.0) 66 : cluster [WRN] Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T07:20:02.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:01 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:59.976338+0000 mgr.y (mgr.44106) 15 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-17T07:20:02.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:01 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:19:59.979966+0000 mgr.y (mgr.44106) 16 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:20:02.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:01 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:19:59.980543+0000 mgr.y (mgr.44106) 17 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-17T07:20:02.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:01 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:20:00.128978+0000 mgr.y (mgr.44106) 18 : cephadm [INF] Upgrade: It appears safe to stop mon.c 2024-09-17T07:20:02.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:01 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:20:00.975065+0000 mon.a (mon.0) 66 : cluster [WRN] Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T07:20:04.044 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:03 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: cluster 2024-09-17T07:20:01.981410+0000 mgr.y (mgr.44106) 19 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T07:20:04.044 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:03 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:20:03.020274+0000 mon.a (mon.0) 67 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:04.044 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:03 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:20:03.021160+0000 mon.a (mon.0) 68 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T07:20:04.044 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:03 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:20:03.021793+0000 mon.a (mon.0) 69 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T07:20:04.044 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:03 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: audit 2024-09-17T07:20:03.022430+0000 mon.a (mon.0) 70 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:04.315 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:03 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:20:01.981410+0000 mgr.y (mgr.44106) 19 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T07:20:04.315 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:03 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:20:03.020274+0000 mon.a (mon.0) 67 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:04.315 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:03 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:20:03.021160+0000 mon.a (mon.0) 68 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T07:20:04.315 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:03 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:20:03.021793+0000 mon.a (mon.0) 69 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T07:20:04.315 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:03 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:20:03.022430+0000 mon.a (mon.0) 70 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:04.685 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:04 smithi044 systemd[1]: Stopping Ceph mon.c for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:20:05.022 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:04 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: debug 2024-09-17T07:20:04.830+0000 7f0fd1c51700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-09-17T07:20:05.022 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:04 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c[26937]: debug 2024-09-17T07:20:04.830+0000 7f0fd1c51700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-09-17T07:20:05.324 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:05 smithi044 podman[60184]: 2024-09-17 07:20:05.024834854 +0000 UTC m=+0.299803294 container died e0d29cfe87a215cce303a92b0c5683f4569407291aab8de569689c832acea20a (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c, GIT_CLEAN=True, maintainer=Dimitri Savineau , org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=HEAD, ceph=True, org.label-schema.build-date=20201204, CEPH_POINT_RELEASE=-16.2.0, GIT_BRANCH=HEAD, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, org.label-schema.name=CentOS Base Image, org.label-schema.vendor=CentOS) 2024-09-17T07:20:05.665 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:05 smithi044 podman[60184]: 2024-09-17 07:20:05.414215013 +0000 UTC m=+0.689183447 container cleanup e0d29cfe87a215cce303a92b0c5683f4569407291aab8de569689c832acea20a (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Dimitri Savineau , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=-16.2.0, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, RELEASE=HEAD, ceph=True, org.label-schema.name=CentOS Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20201204) 2024-09-17T07:20:05.666 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:05 smithi044 bash[60184]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c 2024-09-17T07:20:05.944 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:05 smithi044 podman[60196]: 2024-09-17 07:20:05.665703269 +0000 UTC m=+0.631323067 container remove e0d29cfe87a215cce303a92b0c5683f4569407291aab8de569689c832acea20a (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.c, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Dimitri Savineau , org.label-schema.build-date=20201204, CEPH_POINT_RELEASE=-16.2.0, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, ceph=True, GIT_BRANCH=HEAD) 2024-09-17T07:20:06.324 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:05 smithi044 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@mon.c.service: Deactivated successfully. 2024-09-17T07:20:06.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:05 smithi044 systemd[1]: Stopped Ceph mon.c for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:20:06.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 systemd[1]: Starting Ceph mon.c for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:20:06.661 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 podman[60337]: 2024-09-17 07:20:06.293051547 +0000 UTC m=+0.016222332 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:20:06.661 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 podman[60337]: 2024-09-17 07:20:06.41734035 +0000 UTC m=+0.140511119 container create 02f85d31e0bd0c47b0f5496a08fe10a8831fdf9a1fe2550f404c83d3c87434f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon-c, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-17T07:20:06.661 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 podman[60337]: 2024-09-17 07:20:06.661463362 +0000 UTC m=+0.384634138 container init 02f85d31e0bd0c47b0f5496a08fe10a8831fdf9a1fe2550f404c83d3c87434f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon-c, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b) 2024-09-17T07:20:07.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 podman[60337]: 2024-09-17 07:20:06.664912404 +0000 UTC m=+0.388083171 container start 02f85d31e0bd0c47b0f5496a08fe10a8831fdf9a1fe2550f404c83d3c87434f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon-c, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, ceph=True, GIT_BRANCH=HEAD) 2024-09-17T07:20:07.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: set uid:gid to 167:167 (ceph:ceph) 2024-09-17T07:20:07.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable), process ceph-mon, pid 2 2024-09-17T07:20:07.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: pidfile_write: ignore empty --pid-file 2024-09-17T07:20:07.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: load: jerasure load: lrc 2024-09-17T07:20:07.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: RocksDB version: 7.9.2 2024-09-17T07:20:07.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Git sha 0 2024-09-17T07:20:07.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Compile date 2024-09-16 15:15:13 2024-09-17T07:20:07.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: DB SUMMARY 2024-09-17T07:20:07.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: DB Session ID: UZ11SQU1X722TJGL380K 2024-09-17T07:20:07.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: CURRENT file: CURRENT 2024-09-17T07:20:07.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: IDENTITY file: IDENTITY 2024-09-17T07:20:07.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: MANIFEST file: MANIFEST-000005 size: 896 Bytes 2024-09-17T07:20:07.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000020.sst 2024-09-17T07:20:07.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000018.log size: 6264150 ; 2024-09-17T07:20:07.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.error_if_exists: 0 2024-09-17T07:20:07.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.create_if_missing: 0 2024-09-17T07:20:07.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.paranoid_checks: 1 2024-09-17T07:20:07.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-17T07:20:07.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-17T07:20:07.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-17T07:20:07.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.env: 0x556acde0bbc0 2024-09-17T07:20:07.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.fs: PosixFileSystem 2024-09-17T07:20:07.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.info_log: 0x556ace6eabe0 2024-09-17T07:20:07.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.max_file_opening_threads: 16 2024-09-17T07:20:07.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.statistics: (nil) 2024-09-17T07:20:07.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.use_fsync: 0 2024-09-17T07:20:07.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.max_log_file_size: 0 2024-09-17T07:20:07.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-17T07:20:07.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-17T07:20:07.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.keep_log_file_num: 1000 2024-09-17T07:20:07.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.recycle_log_file_num: 0 2024-09-17T07:20:07.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.allow_fallocate: 1 2024-09-17T07:20:07.078 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.allow_mmap_reads: 0 2024-09-17T07:20:07.078 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.allow_mmap_writes: 0 2024-09-17T07:20:07.078 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.use_direct_reads: 0 2024-09-17T07:20:07.078 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-17T07:20:07.079 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.create_missing_column_families: 0 2024-09-17T07:20:07.079 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.db_log_dir: 2024-09-17T07:20:07.079 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.wal_dir: 2024-09-17T07:20:07.079 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-17T07:20:07.079 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-17T07:20:07.079 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-17T07:20:07.079 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-17T07:20:07.079 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-17T07:20:07.079 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-17T07:20:07.079 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.advise_random_on_open: 1 2024-09-17T07:20:07.079 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.db_write_buffer_size: 0 2024-09-17T07:20:07.079 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.write_buffer_manager: 0x556ace6f8b40 2024-09-17T07:20:07.079 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-17T07:20:07.079 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-17T07:20:07.080 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-17T07:20:07.080 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.rate_limiter: (nil) 2024-09-17T07:20:07.080 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-17T07:20:07.080 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.wal_recovery_mode: 2 2024-09-17T07:20:07.080 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.enable_thread_tracking: 0 2024-09-17T07:20:07.080 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.enable_pipelined_write: 0 2024-09-17T07:20:07.080 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.unordered_write: 0 2024-09-17T07:20:07.080 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-17T07:20:07.080 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-17T07:20:07.080 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-17T07:20:07.080 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-17T07:20:07.080 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.row_cache: None 2024-09-17T07:20:07.080 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.wal_filter: None 2024-09-17T07:20:07.081 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-17T07:20:07.081 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.allow_ingest_behind: 0 2024-09-17T07:20:07.081 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.two_write_queues: 0 2024-09-17T07:20:07.081 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.manual_wal_flush: 0 2024-09-17T07:20:07.081 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.wal_compression: 0 2024-09-17T07:20:07.081 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.atomic_flush: 0 2024-09-17T07:20:07.081 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-17T07:20:07.081 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-17T07:20:07.081 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-17T07:20:07.081 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.log_readahead_size: 0 2024-09-17T07:20:07.081 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-17T07:20:07.081 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.best_efforts_recovery: 0 2024-09-17T07:20:07.081 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-17T07:20:07.081 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-17T07:20:07.082 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.allow_data_in_errors: 0 2024-09-17T07:20:07.082 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.db_host_id: __hostname__ 2024-09-17T07:20:07.082 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-17T07:20:07.082 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.max_background_jobs: 2 2024-09-17T07:20:07.082 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.max_background_compactions: -1 2024-09-17T07:20:07.082 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.max_subcompactions: 1 2024-09-17T07:20:07.082 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-17T07:20:07.082 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-17T07:20:07.082 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-17T07:20:07.082 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.max_total_wal_size: 0 2024-09-17T07:20:07.082 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-17T07:20:07.082 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-17T07:20:07.082 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-17T07:20:07.082 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-17T07:20:07.083 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.max_open_files: -1 2024-09-17T07:20:07.083 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.bytes_per_sync: 0 2024-09-17T07:20:07.083 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-17T07:20:07.083 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-17T07:20:07.083 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.compaction_readahead_size: 0 2024-09-17T07:20:07.083 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.max_background_flushes: -1 2024-09-17T07:20:07.083 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Compression algorithms supported: 2024-09-17T07:20:07.083 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: kZSTD supported: 0 2024-09-17T07:20:07.083 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: kXpressCompression supported: 0 2024-09-17T07:20:07.083 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: kBZip2Compression supported: 0 2024-09-17T07:20:07.083 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-17T07:20:07.083 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: kLZ4Compression supported: 1 2024-09-17T07:20:07.083 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: kZlibCompression supported: 1 2024-09-17T07:20:07.083 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: kLZ4HCCompression supported: 1 2024-09-17T07:20:07.084 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: kSnappyCompression supported: 1 2024-09-17T07:20:07.084 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-17T07:20:07.084 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-17T07:20:07.084 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-09-17T07:20:07.084 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-17T07:20:07.084 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-17T07:20:07.084 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.merge_operator: 2024-09-17T07:20:07.084 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.compaction_filter: None 2024-09-17T07:20:07.084 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.compaction_filter_factory: None 2024-09-17T07:20:07.084 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.sst_partitioner_factory: None 2024-09-17T07:20:07.084 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-17T07:20:07.084 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-17T07:20:07.084 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556ace6ea7e0) 2024-09-17T07:20:07.084 INFO:journalctl@ceph.mon.c.smithi044.stdout: cache_index_and_filter_blocks: 1 2024-09-17T07:20:07.085 INFO:journalctl@ceph.mon.c.smithi044.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-17T07:20:07.085 INFO:journalctl@ceph.mon.c.smithi044.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-17T07:20:07.085 INFO:journalctl@ceph.mon.c.smithi044.stdout: pin_top_level_index_and_filter: 1 2024-09-17T07:20:07.085 INFO:journalctl@ceph.mon.c.smithi044.stdout: index_type: 0 2024-09-17T07:20:07.085 INFO:journalctl@ceph.mon.c.smithi044.stdout: data_block_index_type: 0 2024-09-17T07:20:07.085 INFO:journalctl@ceph.mon.c.smithi044.stdout: index_shortening: 1 2024-09-17T07:20:07.085 INFO:journalctl@ceph.mon.c.smithi044.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-17T07:20:07.085 INFO:journalctl@ceph.mon.c.smithi044.stdout: checksum: 4 2024-09-17T07:20:07.085 INFO:journalctl@ceph.mon.c.smithi044.stdout: no_block_cache: 0 2024-09-17T07:20:07.085 INFO:journalctl@ceph.mon.c.smithi044.stdout: block_cache: 0x556ace6df350 2024-09-17T07:20:07.085 INFO:journalctl@ceph.mon.c.smithi044.stdout: block_cache_name: BinnedLRUCache 2024-09-17T07:20:07.085 INFO:journalctl@ceph.mon.c.smithi044.stdout: block_cache_options: 2024-09-17T07:20:07.085 INFO:journalctl@ceph.mon.c.smithi044.stdout: capacity : 536870912 2024-09-17T07:20:07.085 INFO:journalctl@ceph.mon.c.smithi044.stdout: num_shard_bits : 4 2024-09-17T07:20:07.086 INFO:journalctl@ceph.mon.c.smithi044.stdout: strict_capacity_limit : 0 2024-09-17T07:20:07.086 INFO:journalctl@ceph.mon.c.smithi044.stdout: high_pri_pool_ratio: 0.000 2024-09-17T07:20:07.086 INFO:journalctl@ceph.mon.c.smithi044.stdout: block_cache_compressed: (nil) 2024-09-17T07:20:07.086 INFO:journalctl@ceph.mon.c.smithi044.stdout: persistent_cache: (nil) 2024-09-17T07:20:07.086 INFO:journalctl@ceph.mon.c.smithi044.stdout: block_size: 4096 2024-09-17T07:20:07.086 INFO:journalctl@ceph.mon.c.smithi044.stdout: block_size_deviation: 10 2024-09-17T07:20:07.086 INFO:journalctl@ceph.mon.c.smithi044.stdout: block_restart_interval: 16 2024-09-17T07:20:07.086 INFO:journalctl@ceph.mon.c.smithi044.stdout: index_block_restart_interval: 1 2024-09-17T07:20:07.086 INFO:journalctl@ceph.mon.c.smithi044.stdout: metadata_block_size: 4096 2024-09-17T07:20:07.086 INFO:journalctl@ceph.mon.c.smithi044.stdout: partition_filters: 0 2024-09-17T07:20:07.086 INFO:journalctl@ceph.mon.c.smithi044.stdout: use_delta_encoding: 1 2024-09-17T07:20:07.086 INFO:journalctl@ceph.mon.c.smithi044.stdout: filter_policy: bloomfilter 2024-09-17T07:20:07.086 INFO:journalctl@ceph.mon.c.smithi044.stdout: whole_key_filtering: 1 2024-09-17T07:20:07.086 INFO:journalctl@ceph.mon.c.smithi044.stdout: verify_compression: 0 2024-09-17T07:20:07.087 INFO:journalctl@ceph.mon.c.smithi044.stdout: read_amp_bytes_per_bit: 0 2024-09-17T07:20:07.087 INFO:journalctl@ceph.mon.c.smithi044.stdout: format_version: 5 2024-09-17T07:20:07.087 INFO:journalctl@ceph.mon.c.smithi044.stdout: enable_index_compression: 1 2024-09-17T07:20:07.087 INFO:journalctl@ceph.mon.c.smithi044.stdout: block_align: 0 2024-09-17T07:20:07.087 INFO:journalctl@ceph.mon.c.smithi044.stdout: max_auto_readahead_size: 262144 2024-09-17T07:20:07.087 INFO:journalctl@ceph.mon.c.smithi044.stdout: prepopulate_block_cache: 0 2024-09-17T07:20:07.087 INFO:journalctl@ceph.mon.c.smithi044.stdout: initial_auto_readahead_size: 8192 2024-09-17T07:20:07.087 INFO:journalctl@ceph.mon.c.smithi044.stdout: num_file_reads_for_auto_readahead: 2 2024-09-17T07:20:07.087 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.write_buffer_size: 33554432 2024-09-17T07:20:07.087 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.max_write_buffer_number: 2 2024-09-17T07:20:07.087 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.compression: NoCompression 2024-09-17T07:20:07.087 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.bottommost_compression: Disabled 2024-09-17T07:20:07.087 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.prefix_extractor: nullptr 2024-09-17T07:20:07.087 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-17T07:20:07.088 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.num_levels: 7 2024-09-17T07:20:07.088 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-17T07:20:07.088 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-17T07:20:07.088 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-17T07:20:07.088 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-17T07:20:07.088 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-17T07:20:07.088 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-17T07:20:07.088 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-17T07:20:07.088 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-17T07:20:07.088 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-17T07:20:07.088 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-17T07:20:07.088 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-17T07:20:07.088 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-17T07:20:07.088 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-17T07:20:07.089 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.compression_opts.level: 32767 2024-09-17T07:20:07.089 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.compression_opts.strategy: 0 2024-09-17T07:20:07.089 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-17T07:20:07.089 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-17T07:20:07.089 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-17T07:20:07.089 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-17T07:20:07.089 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.compression_opts.enabled: false 2024-09-17T07:20:07.089 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-17T07:20:07.089 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-17T07:20:07.089 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-17T07:20:07.089 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-17T07:20:07.089 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.target_file_size_base: 67108864 2024-09-17T07:20:07.089 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-17T07:20:07.089 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-17T07:20:07.090 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-17T07:20:07.090 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-17T07:20:07.090 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-17T07:20:07.090 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-17T07:20:07.090 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-17T07:20:07.090 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-17T07:20:07.090 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-17T07:20:07.090 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-17T07:20:07.090 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-17T07:20:07.090 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-17T07:20:07.090 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-17T07:20:07.090 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-17T07:20:07.090 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.arena_block_size: 1048576 2024-09-17T07:20:07.090 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-17T07:20:07.091 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-17T07:20:07.091 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.disable_auto_compactions: 0 2024-09-17T07:20:07.091 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-17T07:20:07.091 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-17T07:20:07.091 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-17T07:20:07.091 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-17T07:20:07.091 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-17T07:20:07.091 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-17T07:20:07.091 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-17T07:20:07.091 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-17T07:20:07.091 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-17T07:20:07.091 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-17T07:20:07.091 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-17T07:20:07.091 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.inplace_update_support: 0 2024-09-17T07:20:07.092 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-17T07:20:07.092 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-17T07:20:07.092 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-17T07:20:07.092 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-17T07:20:07.092 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.bloom_locality: 0 2024-09-17T07:20:07.092 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.max_successive_merges: 0 2024-09-17T07:20:07.092 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-17T07:20:07.092 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.paranoid_file_checks: 0 2024-09-17T07:20:07.092 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.force_consistency_checks: 1 2024-09-17T07:20:07.092 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.report_bg_io_stats: 0 2024-09-17T07:20:07.092 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.ttl: 2592000 2024-09-17T07:20:07.092 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-17T07:20:07.092 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-17T07:20:07.093 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-17T07:20:07.093 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.enable_blob_files: false 2024-09-17T07:20:07.093 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.min_blob_size: 0 2024-09-17T07:20:07.093 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.blob_file_size: 268435456 2024-09-17T07:20:07.093 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-17T07:20:07.093 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-17T07:20:07.093 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-17T07:20:07.093 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-17T07:20:07.093 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-17T07:20:07.093 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.blob_file_starting_level: 0 2024-09-17T07:20:07.093 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-17T07:20:07.093 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 20.sst 2024-09-17T07:20:07.093 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-17T07:20:07.093 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 22, last_sequence is 12057, log_number is 18,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-17T07:20:07.094 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 18 2024-09-17T07:20:07.094 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 8c973802-8fee-4592-a273-e69a582ef13c 2024-09-17T07:20:07.094 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726557606691832, "job": 1, "event": "recovery_started", "wal_files": [18]} 2024-09-17T07:20:07.094 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #18 mode 2 2024-09-17T07:20:07.094 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726557606707091, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 23, "file_size": 4064662, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12054, "largest_seqno": 12730, "table_properties": {"data_size": 4060075, "index_size": 2966, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 4881, "raw_average_key_size": 21, "raw_value_size": 4053456, "raw_average_value_size": 17935, "num_data_blocks": 133, "num_entries": 226, "num_filter_entries": 226, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726557606, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "8c973802-8fee-4592-a273-e69a582ef13c", "db_session_id": "UZ11SQU1X722TJGL380K", "orig_file_number": 23, "seqno_to_time_mapping": "N/A"}} 2024-09-17T07:20:07.094 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726557606707229, "job": 1, "event": "recovery_finished"} 2024-09-17T07:20:07.094 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: [db/version_set.cc:5047] Creating manifest 25 2024-09-17T07:20:07.094 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-17T07:20:07.094 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000018.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-17T07:20:07.094 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x556ace70ae00 2024-09-17T07:20:07.094 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: DB pointer 0x556ace7fe000 2024-09-17T07:20:07.094 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-17T07:20:07.094 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-17T07:20:07.095 INFO:journalctl@ceph.mon.c.smithi044.stdout: ** DB Stats ** 2024-09-17T07:20:07.095 INFO:journalctl@ceph.mon.c.smithi044.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T07:20:07.095 INFO:journalctl@ceph.mon.c.smithi044.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-17T07:20:07.095 INFO:journalctl@ceph.mon.c.smithi044.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T07:20:07.095 INFO:journalctl@ceph.mon.c.smithi044.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T07:20:07.095 INFO:journalctl@ceph.mon.c.smithi044.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-17T07:20:07.095 INFO:journalctl@ceph.mon.c.smithi044.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T07:20:07.095 INFO:journalctl@ceph.mon.c.smithi044.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T07:20:07.095 INFO:journalctl@ceph.mon.c.smithi044.stdout: 2024-09-17T07:20:07.095 INFO:journalctl@ceph.mon.c.smithi044.stdout: ** Compaction Stats [default] ** 2024-09-17T07:20:07.095 INFO:journalctl@ceph.mon.c.smithi044.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T07:20:07.095 INFO:journalctl@ceph.mon.c.smithi044.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-17T07:20:07.095 INFO:journalctl@ceph.mon.c.smithi044.stdout: L0 1/0 3.88 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 526.2 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-09-17T07:20:07.095 INFO:journalctl@ceph.mon.c.smithi044.stdout: L6 1/0 24.60 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-17T07:20:07.096 INFO:journalctl@ceph.mon.c.smithi044.stdout: Sum 2/0 28.48 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 526.2 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-09-17T07:20:07.096 INFO:journalctl@ceph.mon.c.smithi044.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 526.2 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-09-17T07:20:07.096 INFO:journalctl@ceph.mon.c.smithi044.stdout: 2024-09-17T07:20:07.096 INFO:journalctl@ceph.mon.c.smithi044.stdout: ** Compaction Stats [default] ** 2024-09-17T07:20:07.096 INFO:journalctl@ceph.mon.c.smithi044.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T07:20:07.096 INFO:journalctl@ceph.mon.c.smithi044.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T07:20:07.096 INFO:journalctl@ceph.mon.c.smithi044.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 526.2 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-09-17T07:20:07.096 INFO:journalctl@ceph.mon.c.smithi044.stdout: 2024-09-17T07:20:07.096 INFO:journalctl@ceph.mon.c.smithi044.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-17T07:20:07.096 INFO:journalctl@ceph.mon.c.smithi044.stdout: 2024-09-17T07:20:07.096 INFO:journalctl@ceph.mon.c.smithi044.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T07:20:07.096 INFO:journalctl@ceph.mon.c.smithi044.stdout: Flush(GB): cumulative 0.004, interval 0.004 2024-09-17T07:20:07.096 INFO:journalctl@ceph.mon.c.smithi044.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T07:20:07.096 INFO:journalctl@ceph.mon.c.smithi044.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T07:20:07.097 INFO:journalctl@ceph.mon.c.smithi044.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T07:20:07.097 INFO:journalctl@ceph.mon.c.smithi044.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-17T07:20:07.097 INFO:journalctl@ceph.mon.c.smithi044.stdout: Cumulative compaction: 0.00 GB write, 215.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T07:20:07.097 INFO:journalctl@ceph.mon.c.smithi044.stdout: Interval compaction: 0.00 GB write, 215.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T07:20:07.097 INFO:journalctl@ceph.mon.c.smithi044.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T07:20:07.097 INFO:journalctl@ceph.mon.c.smithi044.stdout: Block cache BinnedLRUCache@0x556ace6df350#2 capacity: 512.00 MB usage: 3.73 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-09-17T07:20:07.097 INFO:journalctl@ceph.mon.c.smithi044.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.66 KB,0.00012517%) IndexBlock(1,3.08 KB,0.000587106%) Misc(1,0.00 KB,0%) 2024-09-17T07:20:07.097 INFO:journalctl@ceph.mon.c.smithi044.stdout: 2024-09-17T07:20:07.097 INFO:journalctl@ceph.mon.c.smithi044.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-17T07:20:07.097 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: starting mon.c rank 1 at public addrs [v2:172.21.15.44:3301/0,v1:172.21.15.44:6790/0] at bind addrs [v2:172.21.15.44:3301/0,v1:172.21.15.44:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 817ae996-74c3-11ef-bceb-c7b262605968 2024-09-17T07:20:07.097 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: mon.c@-1(???) e3 preinit fsid 817ae996-74c3-11ef-bceb-c7b262605968 2024-09-17T07:20:07.097 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: mon.c@-1(???).mds e1 new map 2024-09-17T07:20:07.097 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: mon.c@-1(???).mds e1 print_map 2024-09-17T07:20:07.097 INFO:journalctl@ceph.mon.c.smithi044.stdout: e1 2024-09-17T07:20:07.097 INFO:journalctl@ceph.mon.c.smithi044.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-17T07:20:07.098 INFO:journalctl@ceph.mon.c.smithi044.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-17T07:20:07.098 INFO:journalctl@ceph.mon.c.smithi044.stdout: legacy client fscid: -1 2024-09-17T07:20:07.098 INFO:journalctl@ceph.mon.c.smithi044.stdout: 2024-09-17T07:20:07.098 INFO:journalctl@ceph.mon.c.smithi044.stdout: No filesystems configured 2024-09-17T07:20:07.098 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: mon.c@-1(???).osd e76 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-17T07:20:07.098 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: mon.c@-1(???).osd e76 crush map has features 432629239337189376, adjusting msgr requires 2024-09-17T07:20:07.098 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: mon.c@-1(???).osd e76 crush map has features 432629239337189376, adjusting msgr requires 2024-09-17T07:20:07.098 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: mon.c@-1(???).osd e76 crush map has features 432629239337189376, adjusting msgr requires 2024-09-17T07:20:07.098 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 ceph-mon[60356]: mon.c@-1(???).paxosservice(auth 1..24) refresh upgraded, format 0 -> 3 2024-09-17T07:20:07.098 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 bash[60337]: 02f85d31e0bd0c47b0f5496a08fe10a8831fdf9a1fe2550f404c83d3c87434f3 2024-09-17T07:20:07.098 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:06 smithi044 systemd[1]: Started Ceph mon.c for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:20:08.688 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Scheduled restart job, restart counter is at 4. 2024-09-17T07:20:08.688 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 systemd[1]: Stopped Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:20:08.688 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 systemd[1]: Starting Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:20:08.689 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 podman[49454]: 2024-09-17 07:20:08.45709532 +0000 UTC m=+0.018067493 image pull de242295e2257c37c8cadfd962369228f8f10b2d48a44259b65fef44ad4f6490 docker.io/prom/prometheus:v2.18.1 2024-09-17T07:20:08.689 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 podman[49454]: 2024-09-17 07:20:08.569549816 +0000 UTC m=+0.130521975 container create db7aa5b31420aae953e394997bfdea1bd1677f7157c699fdb9d3cf6df0456cf9 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:20:09.114 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 podman[49454]: 2024-09-17 07:20:08.796322972 +0000 UTC m=+0.357295131 container init db7aa5b31420aae953e394997bfdea1bd1677f7157c699fdb9d3cf6df0456cf9 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:20:09.114 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 podman[49454]: 2024-09-17 07:20:08.799550386 +0000 UTC m=+0.360522545 container start db7aa5b31420aae953e394997bfdea1bd1677f7157c699fdb9d3cf6df0456cf9 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:20:09.114 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49468]: level=info ts=2024-09-17T07:20:08.830Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-17T07:20:09.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49468]: level=info ts=2024-09-17T07:20:08.830Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2024-09-17T07:20:09.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49468]: level=info ts=2024-09-17T07:20:08.830Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2024-09-17T07:20:09.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49468]: level=info ts=2024-09-17T07:20:08.830Z caller=main.go:339 host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi179 (none))" 2024-09-17T07:20:09.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49468]: level=info ts=2024-09-17T07:20:08.830Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T07:20:09.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49468]: level=info ts=2024-09-17T07:20:08.830Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T07:20:09.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49468]: level=info ts=2024-09-17T07:20:08.831Z caller=main.go:678 msg="Starting TSDB ..." 2024-09-17T07:20:09.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49468]: level=info ts=2024-09-17T07:20:08.831Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2024-09-17T07:20:09.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49468]: level=info ts=2024-09-17T07:20:08.833Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2024-09-17T07:20:09.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49468]: level=info ts=2024-09-17T07:20:08.834Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=11 2024-09-17T07:20:09.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49468]: level=info ts=2024-09-17T07:20:08.840Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=11 2024-09-17T07:20:09.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49468]: level=info ts=2024-09-17T07:20:08.840Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=11 2024-09-17T07:20:09.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49468]: level=info ts=2024-09-17T07:20:08.840Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=11 2024-09-17T07:20:09.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49468]: level=info ts=2024-09-17T07:20:08.840Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=11 2024-09-17T07:20:09.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49468]: level=info ts=2024-09-17T07:20:08.841Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=11 2024-09-17T07:20:09.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:20:03.981900+0000 mgr.y (mgr.44106) 22 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T07:20:09.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:20:05.982511+0000 mgr.y (mgr.44106) 23 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T07:20:09.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:20:06.918849+0000 mon.c (mon.1) 1 : cluster [INF] mon.c calling monitor election 2024-09-17T07:20:09.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:20:07.893347+0000 mon.a (mon.0) 71 : cluster [INF] mon.a calling monitor election 2024-09-17T07:20:09.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:20:07.896022+0000 mon.a (mon.0) 72 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T07:20:09.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:20:07.901502+0000 mon.a (mon.0) 73 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0],b=[v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0],c=[v2:172.21.15.44:3301/0,v1:172.21.15.44:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T07:20:09.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:20:07.902143+0000 mon.a (mon.0) 74 : cluster [DBG] fsmap 2024-09-17T07:20:09.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:20:07.902168+0000 mon.a (mon.0) 75 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-09-17T07:20:09.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:20:07.903397+0000 mon.a (mon.0) 76 : cluster [DBG] mgrmap e31: y(active, since 17s), standbys: x 2024-09-17T07:20:09.117 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:20:07.903691+0000 mon.a (mon.0) 77 : cluster [WRN] Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-09-17T07:20:09.117 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:20:07.903727+0000 mon.a (mon.0) 78 : cluster [WRN] [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T07:20:09.117 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:20:07.903745+0000 mon.a (mon.0) 79 : cluster [WRN] daemon iscsi.foo.smithi044.elmtfk on smithi044 is in unknown state 2024-09-17T07:20:09.117 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:20:07.903765+0000 mon.a (mon.0) 80 : cluster [WRN] daemon prometheus.a on smithi179 is in unknown state 2024-09-17T07:20:09.117 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:20:07.910194+0000 mon.a (mon.0) 81 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:09.117 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:20:07.915233+0000 mon.a (mon.0) 82 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:09.117 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:20:07.916546+0000 mon.a (mon.0) 83 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:20:09.117 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:20:07.982863+0000 mgr.y (mgr.44106) 24 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-09-17T07:20:09.117 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49468]: level=info ts=2024-09-17T07:20:08.841Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=11 2024-09-17T07:20:09.118 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49468]: level=info ts=2024-09-17T07:20:08.841Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=7 maxSegment=11 2024-09-17T07:20:09.118 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49468]: level=info ts=2024-09-17T07:20:08.841Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=8 maxSegment=11 2024-09-17T07:20:09.118 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49468]: level=info ts=2024-09-17T07:20:08.841Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=9 maxSegment=11 2024-09-17T07:20:09.118 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49468]: level=info ts=2024-09-17T07:20:08.842Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=10 maxSegment=11 2024-09-17T07:20:09.118 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49468]: level=info ts=2024-09-17T07:20:08.842Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=11 maxSegment=11 2024-09-17T07:20:09.118 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49468]: level=info ts=2024-09-17T07:20:08.842Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=8.532965ms 2024-09-17T07:20:09.118 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49468]: level=info ts=2024-09-17T07:20:08.845Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2024-09-17T07:20:09.118 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49468]: level=info ts=2024-09-17T07:20:08.845Z caller=main.go:695 msg="TSDB started" 2024-09-17T07:20:09.118 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49468]: level=info ts=2024-09-17T07:20:08.845Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T07:20:09.118 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49468]: level=info ts=2024-09-17T07:20:08.845Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2024-09-17T07:20:09.118 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49468]: level=info ts=2024-09-17T07:20:08.845Z caller=main.go:561 msg="Stopping notify discovery manager..." 2024-09-17T07:20:09.118 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49468]: level=info ts=2024-09-17T07:20:08.845Z caller=main.go:583 msg="Stopping scrape manager..." 2024-09-17T07:20:09.118 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49468]: level=info ts=2024-09-17T07:20:08.845Z caller=main.go:557 msg="Notify discovery manager stopped" 2024-09-17T07:20:09.118 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49468]: level=info ts=2024-09-17T07:20:08.845Z caller=main.go:543 msg="Scrape discovery manager stopped" 2024-09-17T07:20:09.118 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49468]: level=info ts=2024-09-17T07:20:08.845Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2024-09-17T07:20:09.119 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49468]: level=info ts=2024-09-17T07:20:08.845Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2024-09-17T07:20:09.119 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49468]: level=info ts=2024-09-17T07:20:08.845Z caller=main.go:577 msg="Scrape manager stopped" 2024-09-17T07:20:09.119 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49468]: level=info ts=2024-09-17T07:20:08.845Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2024-09-17T07:20:09.119 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49468]: level=info ts=2024-09-17T07:20:08.845Z caller=main.go:749 msg="Notifier manager stopped" 2024-09-17T07:20:09.119 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49468]: level=error ts=2024-09-17T07:20:08.845Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 14: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain\n line 31: field http_sd_configs not found in type config.plain\n line 45: field http_sd_configs not found in type config.plain\n line 49: field http_sd_configs not found in type config.plain" 2024-09-17T07:20:09.119 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 bash[49454]: db7aa5b31420aae953e394997bfdea1bd1677f7157c699fdb9d3cf6df0456cf9 2024-09-17T07:20:09.119 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 systemd[1]: Started Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:20:09.119 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:08 smithi179 podman[49486]: 2024-09-17 07:20:08.904312491 +0000 UTC m=+0.050853911 container died db7aa5b31420aae953e394997bfdea1bd1677f7157c699fdb9d3cf6df0456cf9 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:20:09.864 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:09 smithi179 podman[49486]: 2024-09-17 07:20:09.478876677 +0000 UTC m=+0.625418093 container remove db7aa5b31420aae953e394997bfdea1bd1677f7157c699fdb9d3cf6df0456cf9 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:20:09.864 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:09 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2024-09-17T07:20:09.864 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:09 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Failed with result 'exit-code'. 2024-09-17T07:20:11.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:11 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:20:09.983276+0000 mgr.y (mgr.44106) 25 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-09-17T07:20:11.978 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:20:11 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:20:11.717+0000 7f5a7ae47640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-17T07:20:14.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:13 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:20:11.983936+0000 mgr.y (mgr.44106) 26 : cluster [DBG] pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:20:14.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:13 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:20:13.516108+0000 mon.c (mon.1) 2 : audit [DBG] from='client.? 172.21.15.44:0/2774548008' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:20:14.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:14 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:20:13.858220+0000 mon.a (mon.0) 84 : audit [INF] from='client.44238 172.21.15.44:0/2800210551' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:20:14.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:14 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:20:14.535316+0000 mon.a (mon.0) 85 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:14.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:14 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:20:14.539682+0000 mon.a (mon.0) 86 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:16.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:20:13.984269+0000 mgr.y (mgr.44106) 27 : cluster [DBG] pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:20:17.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:20:15.985009+0000 mgr.y (mgr.44106) 28 : cluster [DBG] pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:20:17.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:20:16.503235+0000 mon.a (mon.0) 87 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:17.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:20:16.507288+0000 mon.a (mon.0) 88 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:19.800 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:19 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:20:17.985358+0000 mgr.y (mgr.44106) 29 : cluster [DBG] pgmap v18: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:20:19.800 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:19 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Scheduled restart job, restart counter is at 5. 2024-09-17T07:20:19.800 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:19 smithi179 systemd[1]: Stopped Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:20:19.801 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:19 smithi179 systemd[1]: Starting Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:20:20.089 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:19 smithi179 podman[49526]: 2024-09-17 07:20:19.703964864 +0000 UTC m=+0.018200946 image pull de242295e2257c37c8cadfd962369228f8f10b2d48a44259b65fef44ad4f6490 docker.io/prom/prometheus:v2.18.1 2024-09-17T07:20:20.089 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:19 smithi179 podman[49526]: 2024-09-17 07:20:19.813792594 +0000 UTC m=+0.128028674 container create c3062b09b46d89e2eacd2c10b7368802fbfcb41735245c34e521c4af4f73d9d2 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:20:20.089 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 podman[49526]: 2024-09-17 07:20:20.089690168 +0000 UTC m=+0.403926251 container init c3062b09b46d89e2eacd2c10b7368802fbfcb41735245c34e521c4af4f73d9d2 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:20:20.364 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 podman[49526]: 2024-09-17 07:20:20.092924951 +0000 UTC m=+0.407161038 container start c3062b09b46d89e2eacd2c10b7368802fbfcb41735245c34e521c4af4f73d9d2 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:20:20.365 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=info ts=2024-09-17T07:20:20.125Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-17T07:20:20.365 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=info ts=2024-09-17T07:20:20.126Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2024-09-17T07:20:20.365 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=info ts=2024-09-17T07:20:20.126Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2024-09-17T07:20:20.365 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=info ts=2024-09-17T07:20:20.126Z caller=main.go:339 host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi179 (none))" 2024-09-17T07:20:20.365 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=info ts=2024-09-17T07:20:20.126Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T07:20:20.365 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=info ts=2024-09-17T07:20:20.126Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T07:20:20.365 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=info ts=2024-09-17T07:20:20.127Z caller=main.go:678 msg="Starting TSDB ..." 2024-09-17T07:20:20.365 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=info ts=2024-09-17T07:20:20.127Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2024-09-17T07:20:20.365 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=info ts=2024-09-17T07:20:20.130Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2024-09-17T07:20:20.365 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=info ts=2024-09-17T07:20:20.130Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=12 2024-09-17T07:20:20.365 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=info ts=2024-09-17T07:20:20.137Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=12 2024-09-17T07:20:20.365 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=info ts=2024-09-17T07:20:20.137Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=12 2024-09-17T07:20:20.366 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=info ts=2024-09-17T07:20:20.138Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=12 2024-09-17T07:20:20.366 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=info ts=2024-09-17T07:20:20.138Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=12 2024-09-17T07:20:20.366 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=info ts=2024-09-17T07:20:20.138Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=12 2024-09-17T07:20:20.366 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=info ts=2024-09-17T07:20:20.138Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=12 2024-09-17T07:20:20.366 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=info ts=2024-09-17T07:20:20.138Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=7 maxSegment=12 2024-09-17T07:20:20.366 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=info ts=2024-09-17T07:20:20.138Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=8 maxSegment=12 2024-09-17T07:20:20.366 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=info ts=2024-09-17T07:20:20.138Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=9 maxSegment=12 2024-09-17T07:20:20.366 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=info ts=2024-09-17T07:20:20.138Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=10 maxSegment=12 2024-09-17T07:20:20.366 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=info ts=2024-09-17T07:20:20.139Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=11 maxSegment=12 2024-09-17T07:20:20.366 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=info ts=2024-09-17T07:20:20.140Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=12 maxSegment=12 2024-09-17T07:20:20.366 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=info ts=2024-09-17T07:20:20.140Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=9.705322ms 2024-09-17T07:20:20.366 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=info ts=2024-09-17T07:20:20.142Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2024-09-17T07:20:20.366 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=info ts=2024-09-17T07:20:20.142Z caller=main.go:695 msg="TSDB started" 2024-09-17T07:20:20.366 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=info ts=2024-09-17T07:20:20.142Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T07:20:20.367 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=info ts=2024-09-17T07:20:20.143Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2024-09-17T07:20:20.367 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=info ts=2024-09-17T07:20:20.143Z caller=main.go:561 msg="Stopping notify discovery manager..." 2024-09-17T07:20:20.367 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=info ts=2024-09-17T07:20:20.143Z caller=main.go:583 msg="Stopping scrape manager..." 2024-09-17T07:20:20.367 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=info ts=2024-09-17T07:20:20.143Z caller=main.go:557 msg="Notify discovery manager stopped" 2024-09-17T07:20:20.367 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=info ts=2024-09-17T07:20:20.143Z caller=main.go:543 msg="Scrape discovery manager stopped" 2024-09-17T07:20:20.367 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=info ts=2024-09-17T07:20:20.143Z caller=main.go:577 msg="Scrape manager stopped" 2024-09-17T07:20:20.367 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=info ts=2024-09-17T07:20:20.143Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2024-09-17T07:20:20.367 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=info ts=2024-09-17T07:20:20.143Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2024-09-17T07:20:20.367 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=info ts=2024-09-17T07:20:20.143Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2024-09-17T07:20:20.367 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=info ts=2024-09-17T07:20:20.143Z caller=main.go:749 msg="Notifier manager stopped" 2024-09-17T07:20:20.367 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[49574]: level=error ts=2024-09-17T07:20:20.143Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 14: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain\n line 31: field http_sd_configs not found in type config.plain\n line 45: field http_sd_configs not found in type config.plain\n line 49: field http_sd_configs not found in type config.plain" 2024-09-17T07:20:20.367 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 bash[49526]: c3062b09b46d89e2eacd2c10b7368802fbfcb41735245c34e521c4af4f73d9d2 2024-09-17T07:20:20.367 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 systemd[1]: Started Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:20:20.367 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 podman[49670]: 2024-09-17 07:20:20.223293293 +0000 UTC m=+0.072595095 container died c3062b09b46d89e2eacd2c10b7368802fbfcb41735245c34e521c4af4f73d9d2 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:20:20.808 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 podman[49670]: 2024-09-17 07:20:20.790149947 +0000 UTC m=+0.639451735 container remove c3062b09b46d89e2eacd2c10b7368802fbfcb41735245c34e521c4af4f73d9d2 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:20:20.808 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2024-09-17T07:20:21.113 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:20 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Failed with result 'exit-code'. 2024-09-17T07:20:21.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:20:19.821609+0000 mon.a (mon.0) 89 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:21.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:20:19.826243+0000 mon.a (mon.0) 90 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:21.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:20:19.827021+0000 mon.a (mon.0) 91 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:21.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:20:19.827660+0000 mon.a (mon.0) 92 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:20:21.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:20:19.828178+0000 mgr.y (mgr.44106) 30 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:20:21.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:20:19.828607+0000 mgr.y (mgr.44106) 31 : cluster [DBG] pgmap v19: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2024-09-17T07:20:21.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:20:19.828782+0000 mgr.y (mgr.44106) 32 : cluster [DBG] pgmap v20: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2024-09-17T07:20:21.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:20:19.832424+0000 mon.a (mon.0) 93 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:21.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:20:19.946656+0000 mon.a (mon.0) 94 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:20:21.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:20:19.948644+0000 mon.a (mon.0) 95 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:20:21.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:20:19.949567+0000 mon.a (mon.0) 96 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T07:20:21.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:20:19.950110+0000 mon.a (mon.0) 97 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-09-17T07:20:22.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:21 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cephadm 2024-09-17T07:20:19.950493+0000 mgr.y (mgr.44106) 33 : cephadm [INF] Upgrade: It appears safe to stop mon.b 2024-09-17T07:20:23.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: cluster 2024-09-17T07:20:21.829697+0000 mgr.y (mgr.44106) 34 : cluster [DBG] pgmap v21: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2024-09-17T07:20:23.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:20:21.911554+0000 mon.a (mon.0) 98 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:23.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:20:21.912563+0000 mon.a (mon.0) 99 : audit [INF] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T07:20:23.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:20:21.913179+0000 mon.a (mon.0) 100 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T07:20:23.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: audit 2024-09-17T07:20:21.913791+0000 mon.a (mon.0) 101 : audit [DBG] from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:23.368 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-09-17T07:20:23.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:23 smithi179 systemd[1]: Stopping Ceph mon.b for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:20:23.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: debug 2024-09-17T07:20:23.623+0000 7f511910a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-09-17T07:20:23.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b[23922]: debug 2024-09-17T07:20:23.623+0000 7f511910a700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-09-17T07:20:23.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:23 smithi179 podman[49914]: 2024-09-17 07:20:23.726731609 +0000 UTC m=+0.182050389 container died 198f81fe2641001a4b568b31d8be2fe01e4a9ba3304a89c97fd7f3046d3f4619 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b, GIT_CLEAN=True, maintainer=Dimitri Savineau , org.label-schema.build-date=20201204, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-16.2.0, org.label-schema.name=CentOS Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=HEAD) 2024-09-17T07:20:24.036 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T07:20:24.036 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.a smithi044 *:9093,9094 running (2m) 9s ago 6m 16.8M - 0.20.0 0881eb8f169f 0473d4b023ad 2024-09-17T07:20:24.037 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.a smithi179 *:3000 running (99s) 28s ago 6m 27.1M - 6.7.4 557c83e11646 43ab0e03dcac 2024-09-17T07:20:24.037 INFO:teuthology.orchestra.run.smithi044.stdout:iscsi.foo.smithi044.elmtfk smithi044 unknown 9s ago 5m - - 24ecd6d5f14c 17a7859c6333 2024-09-17T07:20:24.037 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.x smithi179 *:8443,8765 running (55s) 28s ago 10m 434M - 18.2.4-855-g19e2a2be 980a0bfcd684 66b52dc26022 2024-09-17T07:20:24.037 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.y smithi044 *:8443,8765 running (92s) 9s ago 12m 471M - 18.2.4-855-g19e2a2be 980a0bfcd684 8080dd318a1c 2024-09-17T07:20:24.037 INFO:teuthology.orchestra.run.smithi044.stdout:mon.a smithi044 running (41s) 9s ago 12m 38.6M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 c3293e8cdbba 2024-09-17T07:20:24.037 INFO:teuthology.orchestra.run.smithi044.stdout:mon.b smithi179 running (11m) 28s ago 11m 49.0M 2048M 16.2.0 24ecd6d5f14c 198f81fe2641 2024-09-17T07:20:24.037 INFO:teuthology.orchestra.run.smithi044.stdout:mon.c smithi044 running (17s) 9s ago 11m 23.5M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 02f85d31e0bd 2024-09-17T07:20:24.037 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.a smithi044 *:9100 running (2m) 9s ago 7m 17.0M - 0.18.1 e5a616e4b9cf 8a2cde5bd9ed 2024-09-17T07:20:24.037 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.b smithi179 *:9100 running (111s) 28s ago 7m 8929k - 0.18.1 e5a616e4b9cf 04c63b5cb524 2024-09-17T07:20:24.037 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (10m) 9s ago 10m 48.4M 4096M 16.2.0 24ecd6d5f14c 4517de6029ac 2024-09-17T07:20:24.037 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (10m) 9s ago 10m 45.8M 4096M 16.2.0 24ecd6d5f14c 2d5122cbeff9 2024-09-17T07:20:24.037 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (9m) 9s ago 9m 43.4M 4096M 16.2.0 24ecd6d5f14c 7fd2f30be874 2024-09-17T07:20:24.037 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi044 running (9m) 9s ago 9m 47.7M 4096M 16.2.0 24ecd6d5f14c 2b7d623395a7 2024-09-17T07:20:24.038 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi179 running (9m) 28s ago 9m 51.5M 4096M 16.2.0 24ecd6d5f14c 911e6f47c822 2024-09-17T07:20:24.038 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi179 running (8m) 28s ago 8m 47.6M 4096M 16.2.0 24ecd6d5f14c e56060742dff 2024-09-17T07:20:24.038 INFO:teuthology.orchestra.run.smithi044.stdout:osd.6 smithi179 running (8m) 28s ago 8m 47.2M 4096M 16.2.0 24ecd6d5f14c 7a5c30b68cbc 2024-09-17T07:20:24.038 INFO:teuthology.orchestra.run.smithi044.stdout:osd.7 smithi179 running (8m) 28s ago 8m 53.3M 4096M 16.2.0 24ecd6d5f14c b0d186fbf8a6 2024-09-17T07:20:24.038 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.a smithi179 *:9095 unknown 28s ago 7m - - 2024-09-17T07:20:24.038 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi044.nhyhnh smithi044 *:8000 running (6m) 9s ago 6m 38.5M - 16.2.0 24ecd6d5f14c 9b455479614f 2024-09-17T07:20:24.038 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi179.tqlzyy smithi179 *:8000 running (6m) 28s ago 6m 40.9M - 16.2.0 24ecd6d5f14c c8d1693241ce 2024-09-17T07:20:24.038 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi044.vnufiz smithi044 *:80 running (6m) 9s ago 6m 38.2M - 16.2.0 24ecd6d5f14c 6eae2bff5ec8 2024-09-17T07:20:24.038 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi179.ecjhll smithi179 *:80 running (6m) 28s ago 6m 41.2M - 16.2.0 24ecd6d5f14c 253109ee0394 2024-09-17T07:20:24.363 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:24 smithi179 podman[49914]: 2024-09-17 07:20:24.073777497 +0000 UTC m=+0.529096266 container cleanup 198f81fe2641001a4b568b31d8be2fe01e4a9ba3304a89c97fd7f3046d3f4619 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b, CEPH_POINT_RELEASE=-16.2.0, maintainer=Dimitri Savineau , org.label-schema.name=CentOS Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.build-date=20201204, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-17T07:20:24.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:24 smithi179 bash[49914]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b 2024-09-17T07:20:24.437 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:20:24.438 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-09-17T07:20:24.438 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 1, 2024-09-17T07:20:24.438 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T07:20:24.438 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:20:24.438 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-09-17T07:20:24.438 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T07:20:24.438 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:20:24.438 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-09-17T07:20:24.438 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2024-09-17T07:20:24.438 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:20:24.439 INFO:teuthology.orchestra.run.smithi044.stdout: "rgw": { 2024-09-17T07:20:24.439 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2024-09-17T07:20:24.439 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:20:24.439 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-09-17T07:20:24.439 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 13, 2024-09-17T07:20:24.439 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 4 2024-09-17T07:20:24.439 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-09-17T07:20:24.439 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:20:24.644 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:24 smithi179 podman[49926]: 2024-09-17 07:20:24.391851232 +0000 UTC m=+0.660928530 container remove 198f81fe2641001a4b568b31d8be2fe01e4a9ba3304a89c97fd7f3046d3f4619 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon.b, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20201204, org.label-schema.schema-version=1.0, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, ceph=True, maintainer=Dimitri Savineau , CEPH_POINT_RELEASE=-16.2.0, RELEASE=HEAD, org.label-schema.name=CentOS Base Image, GIT_BRANCH=HEAD) 2024-09-17T07:20:24.778 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:20:24.778 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9", 2024-09-17T07:20:24.778 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-09-17T07:20:24.778 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T07:20:24.778 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [ 2024-09-17T07:20:24.778 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr" 2024-09-17T07:20:24.778 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-09-17T07:20:24.778 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "4/23 daemons upgraded", 2024-09-17T07:20:24.778 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "Currently upgrading mon daemons", 2024-09-17T07:20:24.778 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-09-17T07:20:24.778 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:20:25.059 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:24 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@mon.b.service: Deactivated successfully. 2024-09-17T07:20:25.059 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:24 smithi179 systemd[1]: Stopped Ceph mon.b for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:20:25.059 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:24 smithi179 systemd[1]: Starting Ceph mon.b for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:20:25.181 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_WARN 2 failed cephadm daemon(s) 2024-09-17T07:20:25.181 INFO:teuthology.orchestra.run.smithi044.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T07:20:25.181 INFO:teuthology.orchestra.run.smithi044.stdout: daemon iscsi.foo.smithi044.elmtfk on smithi044 is in unknown state 2024-09-17T07:20:25.181 INFO:teuthology.orchestra.run.smithi044.stdout: daemon prometheus.a on smithi179 is in unknown state 2024-09-17T07:20:25.321 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 podman[50066]: 2024-09-17 07:20:24.962066957 +0000 UTC m=+0.016004524 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:20:25.322 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 podman[50066]: 2024-09-17 07:20:25.084828265 +0000 UTC m=+0.138765821 container create ade749759184e8bf884522743f9f4c90c1a544720206d79d66053d8099a2a907 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon-b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-17T07:20:25.322 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 podman[50066]: 2024-09-17 07:20:25.321869831 +0000 UTC m=+0.375807396 container init ade749759184e8bf884522743f9f4c90c1a544720206d79d66053d8099a2a907 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon-b, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T07:20:25.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 podman[50066]: 2024-09-17 07:20:25.325112774 +0000 UTC m=+0.379050336 container start ade749759184e8bf884522743f9f4c90c1a544720206d79d66053d8099a2a907 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon-b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T07:20:25.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: set uid:gid to 167:167 (ceph:ceph) 2024-09-17T07:20:25.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable), process ceph-mon, pid 2 2024-09-17T07:20:25.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: pidfile_write: ignore empty --pid-file 2024-09-17T07:20:25.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: load: jerasure load: lrc 2024-09-17T07:20:25.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: RocksDB version: 7.9.2 2024-09-17T07:20:25.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Git sha 0 2024-09-17T07:20:25.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Compile date 2024-09-16 15:15:13 2024-09-17T07:20:25.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: DB SUMMARY 2024-09-17T07:20:25.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: DB Session ID: GK8ARICZI91BC8NEX17F 2024-09-17T07:20:25.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: CURRENT file: CURRENT 2024-09-17T07:20:25.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: IDENTITY file: IDENTITY 2024-09-17T07:20:25.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: MANIFEST file: MANIFEST-000005 size: 896 Bytes 2024-09-17T07:20:25.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000020.sst 2024-09-17T07:20:25.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000018.log size: 7956046 ; 2024-09-17T07:20:25.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.error_if_exists: 0 2024-09-17T07:20:25.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.create_if_missing: 0 2024-09-17T07:20:25.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.paranoid_checks: 1 2024-09-17T07:20:25.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-17T07:20:25.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-17T07:20:25.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-17T07:20:25.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.env: 0x55eefd453bc0 2024-09-17T07:20:25.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.fs: PosixFileSystem 2024-09-17T07:20:25.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.info_log: 0x55eefe626be0 2024-09-17T07:20:25.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.max_file_opening_threads: 16 2024-09-17T07:20:25.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.statistics: (nil) 2024-09-17T07:20:25.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.use_fsync: 0 2024-09-17T07:20:25.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.max_log_file_size: 0 2024-09-17T07:20:25.617 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-17T07:20:25.617 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-17T07:20:25.617 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.keep_log_file_num: 1000 2024-09-17T07:20:25.617 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.recycle_log_file_num: 0 2024-09-17T07:20:25.617 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.allow_fallocate: 1 2024-09-17T07:20:25.617 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.allow_mmap_reads: 0 2024-09-17T07:20:25.617 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.allow_mmap_writes: 0 2024-09-17T07:20:25.617 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.use_direct_reads: 0 2024-09-17T07:20:25.617 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-17T07:20:25.617 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.create_missing_column_families: 0 2024-09-17T07:20:25.617 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.db_log_dir: 2024-09-17T07:20:25.618 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.wal_dir: 2024-09-17T07:20:25.618 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-17T07:20:25.618 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-17T07:20:25.618 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-17T07:20:25.618 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-17T07:20:25.618 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-17T07:20:25.618 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-17T07:20:25.618 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.advise_random_on_open: 1 2024-09-17T07:20:25.618 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.db_write_buffer_size: 0 2024-09-17T07:20:25.618 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.write_buffer_manager: 0x55eefe634b40 2024-09-17T07:20:25.618 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-17T07:20:25.619 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-17T07:20:25.619 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-17T07:20:25.619 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.rate_limiter: (nil) 2024-09-17T07:20:25.619 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-17T07:20:25.619 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.wal_recovery_mode: 2 2024-09-17T07:20:25.619 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.enable_thread_tracking: 0 2024-09-17T07:20:25.619 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.enable_pipelined_write: 0 2024-09-17T07:20:25.619 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.unordered_write: 0 2024-09-17T07:20:25.619 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-17T07:20:25.619 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-17T07:20:25.619 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-17T07:20:25.619 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-17T07:20:25.619 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.row_cache: None 2024-09-17T07:20:25.620 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.wal_filter: None 2024-09-17T07:20:25.620 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-17T07:20:25.620 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.allow_ingest_behind: 0 2024-09-17T07:20:25.620 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.two_write_queues: 0 2024-09-17T07:20:25.620 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.manual_wal_flush: 0 2024-09-17T07:20:25.620 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.wal_compression: 0 2024-09-17T07:20:25.620 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.atomic_flush: 0 2024-09-17T07:20:25.620 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-17T07:20:25.620 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-17T07:20:25.620 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-17T07:20:25.621 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.log_readahead_size: 0 2024-09-17T07:20:25.621 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-17T07:20:25.622 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.best_efforts_recovery: 0 2024-09-17T07:20:25.622 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-17T07:20:25.622 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-17T07:20:25.622 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.allow_data_in_errors: 0 2024-09-17T07:20:25.622 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.db_host_id: __hostname__ 2024-09-17T07:20:25.622 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-17T07:20:25.622 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.max_background_jobs: 2 2024-09-17T07:20:25.622 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.max_background_compactions: -1 2024-09-17T07:20:25.622 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.max_subcompactions: 1 2024-09-17T07:20:25.622 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-17T07:20:25.622 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-17T07:20:25.622 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-17T07:20:25.622 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.max_total_wal_size: 0 2024-09-17T07:20:25.622 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-17T07:20:25.623 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-17T07:20:25.623 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-17T07:20:25.623 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-17T07:20:25.623 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.max_open_files: -1 2024-09-17T07:20:25.623 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.bytes_per_sync: 0 2024-09-17T07:20:25.623 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-17T07:20:25.623 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-17T07:20:25.623 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.compaction_readahead_size: 0 2024-09-17T07:20:25.623 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.max_background_flushes: -1 2024-09-17T07:20:25.623 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Compression algorithms supported: 2024-09-17T07:20:25.623 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: kZSTD supported: 0 2024-09-17T07:20:25.623 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: kXpressCompression supported: 0 2024-09-17T07:20:25.623 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: kBZip2Compression supported: 0 2024-09-17T07:20:25.623 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-17T07:20:25.623 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: kLZ4Compression supported: 1 2024-09-17T07:20:25.624 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: kZlibCompression supported: 1 2024-09-17T07:20:25.624 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: kLZ4HCCompression supported: 1 2024-09-17T07:20:25.624 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: kSnappyCompression supported: 1 2024-09-17T07:20:25.624 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-17T07:20:25.624 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-17T07:20:25.624 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-09-17T07:20:25.624 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-17T07:20:25.624 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-17T07:20:25.624 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.merge_operator: 2024-09-17T07:20:25.624 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.compaction_filter: None 2024-09-17T07:20:25.624 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.compaction_filter_factory: None 2024-09-17T07:20:25.624 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.sst_partitioner_factory: None 2024-09-17T07:20:25.625 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-17T07:20:25.625 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-17T07:20:25.625 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55eefe6267e0) 2024-09-17T07:20:25.625 INFO:journalctl@ceph.mon.b.smithi179.stdout: cache_index_and_filter_blocks: 1 2024-09-17T07:20:25.625 INFO:journalctl@ceph.mon.b.smithi179.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-17T07:20:25.625 INFO:journalctl@ceph.mon.b.smithi179.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-17T07:20:25.625 INFO:journalctl@ceph.mon.b.smithi179.stdout: pin_top_level_index_and_filter: 1 2024-09-17T07:20:25.625 INFO:journalctl@ceph.mon.b.smithi179.stdout: index_type: 0 2024-09-17T07:20:25.625 INFO:journalctl@ceph.mon.b.smithi179.stdout: data_block_index_type: 0 2024-09-17T07:20:25.625 INFO:journalctl@ceph.mon.b.smithi179.stdout: index_shortening: 1 2024-09-17T07:20:25.625 INFO:journalctl@ceph.mon.b.smithi179.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-17T07:20:25.625 INFO:journalctl@ceph.mon.b.smithi179.stdout: checksum: 4 2024-09-17T07:20:25.625 INFO:journalctl@ceph.mon.b.smithi179.stdout: no_block_cache: 0 2024-09-17T07:20:25.625 INFO:journalctl@ceph.mon.b.smithi179.stdout: block_cache: 0x55eefe61b350 2024-09-17T07:20:25.626 INFO:journalctl@ceph.mon.b.smithi179.stdout: block_cache_name: BinnedLRUCache 2024-09-17T07:20:25.626 INFO:journalctl@ceph.mon.b.smithi179.stdout: block_cache_options: 2024-09-17T07:20:25.626 INFO:journalctl@ceph.mon.b.smithi179.stdout: capacity : 536870912 2024-09-17T07:20:25.626 INFO:journalctl@ceph.mon.b.smithi179.stdout: num_shard_bits : 4 2024-09-17T07:20:25.626 INFO:journalctl@ceph.mon.b.smithi179.stdout: strict_capacity_limit : 0 2024-09-17T07:20:25.626 INFO:journalctl@ceph.mon.b.smithi179.stdout: high_pri_pool_ratio: 0.000 2024-09-17T07:20:25.626 INFO:journalctl@ceph.mon.b.smithi179.stdout: block_cache_compressed: (nil) 2024-09-17T07:20:25.626 INFO:journalctl@ceph.mon.b.smithi179.stdout: persistent_cache: (nil) 2024-09-17T07:20:25.626 INFO:journalctl@ceph.mon.b.smithi179.stdout: block_size: 4096 2024-09-17T07:20:25.626 INFO:journalctl@ceph.mon.b.smithi179.stdout: block_size_deviation: 10 2024-09-17T07:20:25.626 INFO:journalctl@ceph.mon.b.smithi179.stdout: block_restart_interval: 16 2024-09-17T07:20:25.626 INFO:journalctl@ceph.mon.b.smithi179.stdout: index_block_restart_interval: 1 2024-09-17T07:20:25.626 INFO:journalctl@ceph.mon.b.smithi179.stdout: metadata_block_size: 4096 2024-09-17T07:20:25.626 INFO:journalctl@ceph.mon.b.smithi179.stdout: partition_filters: 0 2024-09-17T07:20:25.627 INFO:journalctl@ceph.mon.b.smithi179.stdout: use_delta_encoding: 1 2024-09-17T07:20:25.627 INFO:journalctl@ceph.mon.b.smithi179.stdout: filter_policy: bloomfilter 2024-09-17T07:20:25.627 INFO:journalctl@ceph.mon.b.smithi179.stdout: whole_key_filtering: 1 2024-09-17T07:20:25.627 INFO:journalctl@ceph.mon.b.smithi179.stdout: verify_compression: 0 2024-09-17T07:20:25.627 INFO:journalctl@ceph.mon.b.smithi179.stdout: read_amp_bytes_per_bit: 0 2024-09-17T07:20:25.627 INFO:journalctl@ceph.mon.b.smithi179.stdout: format_version: 5 2024-09-17T07:20:25.627 INFO:journalctl@ceph.mon.b.smithi179.stdout: enable_index_compression: 1 2024-09-17T07:20:25.627 INFO:journalctl@ceph.mon.b.smithi179.stdout: block_align: 0 2024-09-17T07:20:25.627 INFO:journalctl@ceph.mon.b.smithi179.stdout: max_auto_readahead_size: 262144 2024-09-17T07:20:25.627 INFO:journalctl@ceph.mon.b.smithi179.stdout: prepopulate_block_cache: 0 2024-09-17T07:20:25.627 INFO:journalctl@ceph.mon.b.smithi179.stdout: initial_auto_readahead_size: 8192 2024-09-17T07:20:25.628 INFO:journalctl@ceph.mon.b.smithi179.stdout: num_file_reads_for_auto_readahead: 2 2024-09-17T07:20:25.628 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.write_buffer_size: 33554432 2024-09-17T07:20:25.628 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.max_write_buffer_number: 2 2024-09-17T07:20:25.628 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.compression: NoCompression 2024-09-17T07:20:25.628 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.bottommost_compression: Disabled 2024-09-17T07:20:25.628 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.prefix_extractor: nullptr 2024-09-17T07:20:25.628 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-17T07:20:25.628 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.num_levels: 7 2024-09-17T07:20:25.628 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-17T07:20:25.629 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-17T07:20:25.629 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-17T07:20:25.629 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-17T07:20:25.629 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-17T07:20:25.629 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-17T07:20:25.629 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-17T07:20:25.629 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-17T07:20:25.629 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-17T07:20:25.629 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-17T07:20:25.629 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-17T07:20:25.629 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-17T07:20:25.629 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-17T07:20:25.629 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.compression_opts.level: 32767 2024-09-17T07:20:25.630 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.compression_opts.strategy: 0 2024-09-17T07:20:25.630 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-17T07:20:25.630 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-17T07:20:25.630 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-17T07:20:25.630 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-17T07:20:25.630 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.compression_opts.enabled: false 2024-09-17T07:20:25.630 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-17T07:20:25.630 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-17T07:20:25.630 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-17T07:20:25.630 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-17T07:20:25.630 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.target_file_size_base: 67108864 2024-09-17T07:20:25.630 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-17T07:20:25.630 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-17T07:20:25.630 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-17T07:20:25.631 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-17T07:20:25.631 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-17T07:20:25.631 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-17T07:20:25.631 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-17T07:20:25.631 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-17T07:20:25.631 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-17T07:20:25.631 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-17T07:20:25.631 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-17T07:20:25.631 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-17T07:20:25.631 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-17T07:20:25.631 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-17T07:20:25.631 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.arena_block_size: 1048576 2024-09-17T07:20:25.631 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-17T07:20:25.632 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-17T07:20:25.632 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.disable_auto_compactions: 0 2024-09-17T07:20:25.632 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-17T07:20:25.632 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-17T07:20:25.632 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-17T07:20:25.632 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-17T07:20:25.632 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-17T07:20:25.632 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-17T07:20:25.632 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-17T07:20:25.632 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-17T07:20:25.632 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-17T07:20:25.632 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-17T07:20:25.632 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-17T07:20:25.633 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.inplace_update_support: 0 2024-09-17T07:20:25.633 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-17T07:20:25.633 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-17T07:20:25.633 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-17T07:20:25.633 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-17T07:20:25.633 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.bloom_locality: 0 2024-09-17T07:20:25.633 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.max_successive_merges: 0 2024-09-17T07:20:25.633 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-17T07:20:25.633 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.paranoid_file_checks: 0 2024-09-17T07:20:25.633 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.force_consistency_checks: 1 2024-09-17T07:20:25.633 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.report_bg_io_stats: 0 2024-09-17T07:20:25.633 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.ttl: 2592000 2024-09-17T07:20:25.633 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-17T07:20:25.633 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-17T07:20:25.634 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-17T07:20:25.634 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.enable_blob_files: false 2024-09-17T07:20:25.634 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.min_blob_size: 0 2024-09-17T07:20:25.634 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.blob_file_size: 268435456 2024-09-17T07:20:25.634 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-17T07:20:25.634 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-17T07:20:25.634 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-17T07:20:25.634 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-17T07:20:25.634 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-17T07:20:25.634 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.blob_file_starting_level: 0 2024-09-17T07:20:25.634 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-17T07:20:25.634 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 20.sst 2024-09-17T07:20:25.634 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-17T07:20:25.634 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 22, last_sequence is 11971, log_number is 18,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-17T07:20:25.635 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 18 2024-09-17T07:20:25.635 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: f6370aef-803a-4848-9871-4d5fc9e1104c 2024-09-17T07:20:25.635 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726557625352192, "job": 1, "event": "recovery_started", "wal_files": [18]} 2024-09-17T07:20:25.635 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #18 mode 2 2024-09-17T07:20:25.635 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726557625369973, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 23, "file_size": 5121574, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11968, "largest_seqno": 12920, "table_properties": {"data_size": 5115581, "index_size": 4180, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 6305, "raw_average_key_size": 20, "raw_value_size": 5106732, "raw_average_value_size": 16965, "num_data_blocks": 186, "num_entries": 301, "num_filter_entries": 301, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726557625, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "f6370aef-803a-4848-9871-4d5fc9e1104c", "db_session_id": "GK8ARICZI91BC8NEX17F", "orig_file_number": 23, "seqno_to_time_mapping": "N/A"}} 2024-09-17T07:20:25.635 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726557625370102, "job": 1, "event": "recovery_finished"} 2024-09-17T07:20:25.635 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: [db/version_set.cc:5047] Creating manifest 25 2024-09-17T07:20:25.635 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-17T07:20:25.635 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000018.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-17T07:20:25.635 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55eefe646e00 2024-09-17T07:20:25.635 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: DB pointer 0x55eefe73a000 2024-09-17T07:20:25.635 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-17T07:20:25.636 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-17T07:20:25.636 INFO:journalctl@ceph.mon.b.smithi179.stdout: ** DB Stats ** 2024-09-17T07:20:25.636 INFO:journalctl@ceph.mon.b.smithi179.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T07:20:25.636 INFO:journalctl@ceph.mon.b.smithi179.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-17T07:20:25.636 INFO:journalctl@ceph.mon.b.smithi179.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T07:20:25.636 INFO:journalctl@ceph.mon.b.smithi179.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T07:20:25.636 INFO:journalctl@ceph.mon.b.smithi179.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-17T07:20:25.636 INFO:journalctl@ceph.mon.b.smithi179.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T07:20:25.636 INFO:journalctl@ceph.mon.b.smithi179.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T07:20:25.636 INFO:journalctl@ceph.mon.b.smithi179.stdout: 2024-09-17T07:20:25.636 INFO:journalctl@ceph.mon.b.smithi179.stdout: ** Compaction Stats [default] ** 2024-09-17T07:20:25.636 INFO:journalctl@ceph.mon.b.smithi179.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T07:20:25.636 INFO:journalctl@ceph.mon.b.smithi179.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-17T07:20:25.636 INFO:journalctl@ceph.mon.b.smithi179.stdout: L0 1/0 4.88 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 581.5 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-09-17T07:20:25.637 INFO:journalctl@ceph.mon.b.smithi179.stdout: L6 1/0 24.60 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-17T07:20:25.637 INFO:journalctl@ceph.mon.b.smithi179.stdout: Sum 2/0 29.49 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 581.5 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-09-17T07:20:25.637 INFO:journalctl@ceph.mon.b.smithi179.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 581.5 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-09-17T07:20:25.637 INFO:journalctl@ceph.mon.b.smithi179.stdout: 2024-09-17T07:20:25.637 INFO:journalctl@ceph.mon.b.smithi179.stdout: ** Compaction Stats [default] ** 2024-09-17T07:20:25.637 INFO:journalctl@ceph.mon.b.smithi179.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T07:20:25.637 INFO:journalctl@ceph.mon.b.smithi179.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T07:20:25.637 INFO:journalctl@ceph.mon.b.smithi179.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 581.5 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-09-17T07:20:25.637 INFO:journalctl@ceph.mon.b.smithi179.stdout: 2024-09-17T07:20:25.637 INFO:journalctl@ceph.mon.b.smithi179.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-17T07:20:25.637 INFO:journalctl@ceph.mon.b.smithi179.stdout: 2024-09-17T07:20:25.637 INFO:journalctl@ceph.mon.b.smithi179.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T07:20:25.637 INFO:journalctl@ceph.mon.b.smithi179.stdout: Flush(GB): cumulative 0.005, interval 0.005 2024-09-17T07:20:25.638 INFO:journalctl@ceph.mon.b.smithi179.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T07:20:25.638 INFO:journalctl@ceph.mon.b.smithi179.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T07:20:25.638 INFO:journalctl@ceph.mon.b.smithi179.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T07:20:25.638 INFO:journalctl@ceph.mon.b.smithi179.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-17T07:20:25.638 INFO:journalctl@ceph.mon.b.smithi179.stdout: Cumulative compaction: 0.00 GB write, 217.28 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T07:20:25.638 INFO:journalctl@ceph.mon.b.smithi179.stdout: Interval compaction: 0.00 GB write, 217.28 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T07:20:25.638 INFO:journalctl@ceph.mon.b.smithi179.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T07:20:25.638 INFO:journalctl@ceph.mon.b.smithi179.stdout: Block cache BinnedLRUCache@0x55eefe61b350#2 capacity: 512.00 MB usage: 5.48 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-09-17T07:20:25.638 INFO:journalctl@ceph.mon.b.smithi179.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.91 KB,0.000172853%) IndexBlock(1,4.58 KB,0.000873208%) Misc(1,0.00 KB,0%) 2024-09-17T07:20:25.639 INFO:journalctl@ceph.mon.b.smithi179.stdout: 2024-09-17T07:20:25.639 INFO:journalctl@ceph.mon.b.smithi179.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-17T07:20:25.639 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: starting mon.b rank 2 at public addrs [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] at bind addrs [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 817ae996-74c3-11ef-bceb-c7b262605968 2024-09-17T07:20:25.639 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: mon.b@-1(???) e3 preinit fsid 817ae996-74c3-11ef-bceb-c7b262605968 2024-09-17T07:20:25.639 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: mon.b@-1(???).mds e1 new map 2024-09-17T07:20:25.639 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: mon.b@-1(???).mds e1 print_map 2024-09-17T07:20:25.639 INFO:journalctl@ceph.mon.b.smithi179.stdout: e1 2024-09-17T07:20:25.639 INFO:journalctl@ceph.mon.b.smithi179.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-17T07:20:25.639 INFO:journalctl@ceph.mon.b.smithi179.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-17T07:20:25.639 INFO:journalctl@ceph.mon.b.smithi179.stdout: legacy client fscid: -1 2024-09-17T07:20:25.640 INFO:journalctl@ceph.mon.b.smithi179.stdout: 2024-09-17T07:20:25.640 INFO:journalctl@ceph.mon.b.smithi179.stdout: No filesystems configured 2024-09-17T07:20:25.640 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: mon.b@-1(???).osd e76 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-17T07:20:25.640 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: mon.b@-1(???).osd e76 crush map has features 432629239337189376, adjusting msgr requires 2024-09-17T07:20:25.640 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: mon.b@-1(???).osd e76 crush map has features 432629239337189376, adjusting msgr requires 2024-09-17T07:20:25.640 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: mon.b@-1(???).osd e76 crush map has features 432629239337189376, adjusting msgr requires 2024-09-17T07:20:25.640 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 ceph-mon[50085]: mon.b@-1(???).paxosservice(auth 1..26) refresh upgraded, format 0 -> 3 2024-09-17T07:20:25.640 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 bash[50066]: ade749759184e8bf884522743f9f4c90c1a544720206d79d66053d8099a2a907 2024-09-17T07:20:25.640 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:25 smithi179 systemd[1]: Started Ceph mon.b for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:20:28.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: mkfs 817ae996-74c3-11ef-bceb-c7b262605968 2024-09-17T07:20:28.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-09-17T07:20:28.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-09-17T07:20:28.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Reconfiguring node-exporter.a (dependencies changed)... 2024-09-17T07:20:28.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Reconfiguring daemon node-exporter.a on smithi044 2024-09-17T07:20:28.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-17T07:20:28.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Reconfiguring daemon alertmanager.a on smithi044 2024-09-17T07:20:28.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Reconfiguring iscsi.foo.smithi044.elmtfk (dependencies changed)... 2024-09-17T07:20:28.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Reconfiguring daemon iscsi.foo.smithi044.elmtfk on smithi044 2024-09-17T07:20:28.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T07:20:28.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Reconfiguring daemon prometheus.a on smithi179 2024-09-17T07:20:28.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Reconfiguring node-exporter.b (dependencies changed)... 2024-09-17T07:20:28.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Reconfiguring daemon node-exporter.b on smithi179 2024-09-17T07:20:28.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Reconfiguring grafana.a (dependencies changed)... 2024-09-17T07:20:28.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-17T07:20:28.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Reconfiguring daemon grafana.a on smithi179 2024-09-17T07:20:28.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T07:20:28.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T07:20:28.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Upgrade: Updating mgr.y 2024-09-17T07:20:28.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Deploying daemon mgr.y on smithi044 2024-09-17T07:20:28.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:20:28.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T07:20:28.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T07:20:28.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Failing over to other MGR 2024-09-17T07:20:28.117 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: inventory: adjusted host smithi179 addr 'smithi179' -> '172.21.15.179' 2024-09-17T07:20:28.117 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: [17/Sep/2024:07:19:06] ENGINE Bus STARTING 2024-09-17T07:20:28.117 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: [17/Sep/2024:07:19:06] ENGINE Serving on https://172.21.15.44:7150 2024-09-17T07:20:28.117 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: [17/Sep/2024:07:19:06] ENGINE Error in HTTPServer.serve 2024-09-17T07:20:28.117 INFO:journalctl@ceph.mon.b.smithi179.stdout: Traceback (most recent call last): 2024-09-17T07:20:28.117 INFO:journalctl@ceph.mon.b.smithi179.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T07:20:28.117 INFO:journalctl@ceph.mon.b.smithi179.stdout: self._connections.run(self.expiration_interval) 2024-09-17T07:20:28.117 INFO:journalctl@ceph.mon.b.smithi179.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T07:20:28.117 INFO:journalctl@ceph.mon.b.smithi179.stdout: self._run(expiration_interval) 2024-09-17T07:20:28.117 INFO:journalctl@ceph.mon.b.smithi179.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T07:20:28.117 INFO:journalctl@ceph.mon.b.smithi179.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T07:20:28.117 INFO:journalctl@ceph.mon.b.smithi179.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T07:20:28.117 INFO:journalctl@ceph.mon.b.smithi179.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T07:20:28.118 INFO:journalctl@ceph.mon.b.smithi179.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T07:20:28.118 INFO:journalctl@ceph.mon.b.smithi179.stdout: s = self.context.wrap_socket( 2024-09-17T07:20:28.118 INFO:journalctl@ceph.mon.b.smithi179.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T07:20:28.118 INFO:journalctl@ceph.mon.b.smithi179.stdout: return self.sslsocket_class._create( 2024-09-17T07:20:28.118 INFO:journalctl@ceph.mon.b.smithi179.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T07:20:28.118 INFO:journalctl@ceph.mon.b.smithi179.stdout: self.do_handshake() 2024-09-17T07:20:28.118 INFO:journalctl@ceph.mon.b.smithi179.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T07:20:28.118 INFO:journalctl@ceph.mon.b.smithi179.stdout: self._sslobj.do_handshake() 2024-09-17T07:20:28.118 INFO:journalctl@ceph.mon.b.smithi179.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T07:20:28.118 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: [17/Sep/2024:07:19:06] ENGINE Serving on http://172.21.15.44:8765 2024-09-17T07:20:28.118 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: [17/Sep/2024:07:19:06] ENGINE Bus STARTED 2024-09-17T07:20:28.118 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:20:28.118 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Reconfiguring iscsi.foo.smithi044.elmtfk (dependencies changed)... 2024-09-17T07:20:28.118 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Reconfiguring daemon iscsi.foo.smithi044.elmtfk on smithi044 2024-09-17T07:20:28.119 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T07:20:28.119 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Reconfiguring daemon prometheus.a on smithi179 2024-09-17T07:20:28.119 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Upgrade: Updating mgr.x 2024-09-17T07:20:28.119 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Deploying daemon mgr.x on smithi179 2024-09-17T07:20:28.119 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:20:28.119 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Upgrade: Setting container_image for all mgr 2024-09-17T07:20:28.119 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Upgrade: It appears safe to stop mon.a 2024-09-17T07:20:28.119 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Upgrade: Updating mon.a 2024-09-17T07:20:28.119 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Deploying daemon mon.a on smithi044 2024-09-17T07:20:28.119 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-09-17T07:20:28.119 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T07:20:28.119 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: daemon iscsi.foo.smithi044.elmtfk on smithi044 is in unknown state 2024-09-17T07:20:28.119 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: daemon prometheus.a on smithi179 is in unknown state 2024-09-17T07:20:28.119 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: mgrmap e25: y(active, since 43s), standbys: x 2024-09-17T07:20:28.119 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: mgrmap e26: y(active, since 44s), standbys: x 2024-09-17T07:20:28.120 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Standby manager daemon x restarted 2024-09-17T07:20:28.120 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Standby manager daemon x started 2024-09-17T07:20:28.120 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: mgrmap e27: y(active, since 47s), standbys: x 2024-09-17T07:20:28.120 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Active manager daemon y restarted 2024-09-17T07:20:28.120 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Activating manager daemon y 2024-09-17T07:20:28.120 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: osdmap e76: 8 total, 8 up, 8 in 2024-09-17T07:20:28.120 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: mgrmap e28: y(active, starting, since 0.0127954s), standbys: x 2024-09-17T07:20:28.120 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Manager daemon y is now available 2024-09-17T07:20:28.120 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:20:28.120 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:20:28.120 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: mgrmap e29: y(active, since 1.01164s), standbys: x 2024-09-17T07:20:28.120 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: [17/Sep/2024:07:19:51] ENGINE Bus STARTING 2024-09-17T07:20:28.120 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: [17/Sep/2024:07:19:51] ENGINE Serving on https://172.21.15.44:7150 2024-09-17T07:20:28.120 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: [17/Sep/2024:07:19:51] ENGINE Error in HTTPServer.serve 2024-09-17T07:20:28.121 INFO:journalctl@ceph.mon.b.smithi179.stdout: Traceback (most recent call last): 2024-09-17T07:20:28.121 INFO:journalctl@ceph.mon.b.smithi179.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T07:20:28.121 INFO:journalctl@ceph.mon.b.smithi179.stdout: self._connections.run(self.expiration_interval) 2024-09-17T07:20:28.121 INFO:journalctl@ceph.mon.b.smithi179.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T07:20:28.121 INFO:journalctl@ceph.mon.b.smithi179.stdout: self._run(expiration_interval) 2024-09-17T07:20:28.121 INFO:journalctl@ceph.mon.b.smithi179.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T07:20:28.121 INFO:journalctl@ceph.mon.b.smithi179.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T07:20:28.121 INFO:journalctl@ceph.mon.b.smithi179.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T07:20:28.121 INFO:journalctl@ceph.mon.b.smithi179.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T07:20:28.121 INFO:journalctl@ceph.mon.b.smithi179.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T07:20:28.121 INFO:journalctl@ceph.mon.b.smithi179.stdout: s = self.context.wrap_socket( 2024-09-17T07:20:28.121 INFO:journalctl@ceph.mon.b.smithi179.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T07:20:28.121 INFO:journalctl@ceph.mon.b.smithi179.stdout: return self.sslsocket_class._create( 2024-09-17T07:20:28.122 INFO:journalctl@ceph.mon.b.smithi179.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T07:20:28.122 INFO:journalctl@ceph.mon.b.smithi179.stdout: self.do_handshake() 2024-09-17T07:20:28.122 INFO:journalctl@ceph.mon.b.smithi179.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T07:20:28.122 INFO:journalctl@ceph.mon.b.smithi179.stdout: self._sslobj.do_handshake() 2024-09-17T07:20:28.122 INFO:journalctl@ceph.mon.b.smithi179.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T07:20:28.122 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2024-09-17T07:20:28.122 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Cluster is now healthy 2024-09-17T07:20:28.122 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: pgmap v4: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:20:28.122 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: mgrmap e30: y(active, since 2s), standbys: x 2024-09-17T07:20:28.122 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: [17/Sep/2024:07:19:52] ENGINE Serving on http://172.21.15.44:8765 2024-09-17T07:20:28.122 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: [17/Sep/2024:07:19:52] ENGINE Bus STARTED 2024-09-17T07:20:28.122 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/2556707789' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:20:28.122 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='client.25351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:20:28.122 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='client.44190 172.21.15.44:0/2380431437' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:20:28.123 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: pgmap v5: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:20:28.123 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: mgrmap e31: y(active, since 4s), standbys: x 2024-09-17T07:20:28.123 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.123 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.123 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.123 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.123 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.123 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.123 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.123 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.124 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: pgmap v6: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:20:28.124 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: pgmap v7: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-17T07:20:28.124 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/3155806008' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:20:28.124 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.125 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: pgmap v8: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-17T07:20:28.125 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.125 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:28.125 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:20:28.125 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:20:28.125 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-17T07:20:28.125 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.125 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: overall HEALTH_OK 2024-09-17T07:20:28.125 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:20:28.125 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:20:28.125 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T07:20:28.125 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-17T07:20:28.125 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Upgrade: It appears safe to stop mon.c 2024-09-17T07:20:28.126 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/1958833622' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2084457509"}]: access denied 2024-09-17T07:20:28.126 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T07:20:28.126 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T07:20:28.126 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.126 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T07:20:28.126 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T07:20:28.126 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:28.126 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T07:20:28.126 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T07:20:28.126 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: mon.c calling monitor election 2024-09-17T07:20:28.126 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: mon.a calling monitor election 2024-09-17T07:20:28.126 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T07:20:28.126 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: monmap e3: 3 mons at {a=[v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0],b=[v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0],c=[v2:172.21.15.44:3301/0,v1:172.21.15.44:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T07:20:28.126 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: fsmap 2024-09-17T07:20:28.126 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: osdmap e76: 8 total, 8 up, 8 in 2024-09-17T07:20:28.127 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: mgrmap e31: y(active, since 17s), standbys: x 2024-09-17T07:20:28.127 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-09-17T07:20:28.127 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T07:20:28.127 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: daemon iscsi.foo.smithi044.elmtfk on smithi044 is in unknown state 2024-09-17T07:20:28.127 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: daemon prometheus.a on smithi179 is in unknown state 2024-09-17T07:20:28.127 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.127 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.127 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:20:28.127 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-09-17T07:20:28.127 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-09-17T07:20:28.127 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:20:28.127 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/2774548008' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:20:28.127 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='client.44238 172.21.15.44:0/2800210551' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:20:28.127 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:20:28.127 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.128 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.128 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:20:28.128 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.128 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.128 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: pgmap v18: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:20:28.128 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.128 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.128 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:28.128 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:20:28.128 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:20:28.128 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: pgmap v19: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2024-09-17T07:20:28.128 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: pgmap v20: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2024-09-17T07:20:28.128 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.128 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:20:28.129 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:20:28.129 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T07:20:28.129 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-09-17T07:20:28.129 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Upgrade: It appears safe to stop mon.b 2024-09-17T07:20:28.129 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: pgmap v21: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2024-09-17T07:20:28.129 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.129 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T07:20:28.129 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T07:20:28.129 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:28.129 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:20:28.129 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: mon.a calling monitor election 2024-09-17T07:20:28.129 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: mon.b calling monitor election 2024-09-17T07:20:28.129 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: mon.c calling monitor election 2024-09-17T07:20:28.130 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:20:28.130 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:20:28.130 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T07:20:28.130 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: monmap e4: 3 mons at {a=[v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0],b=[v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0],c=[v2:172.21.15.44:3301/0,v1:172.21.15.44:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T07:20:28.130 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: fsmap 2024-09-17T07:20:28.130 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: osdmap e76: 8 total, 8 up, 8 in 2024-09-17T07:20:28.130 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: mgrmap e31: y(active, since 36s), standbys: x 2024-09-17T07:20:28.130 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-09-17T07:20:28.130 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T07:20:28.130 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: daemon iscsi.foo.smithi044.elmtfk on smithi044 is in unknown state 2024-09-17T07:20:28.130 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: daemon prometheus.a on smithi179 is in unknown state 2024-09-17T07:20:28.130 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.130 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.130 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:20:28.130 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:27 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/883135932' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:20:28.254 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: mkfs 817ae996-74c3-11ef-bceb-c7b262605968 2024-09-17T07:20:28.254 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-09-17T07:20:28.255 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-09-17T07:20:28.255 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Reconfiguring node-exporter.a (dependencies changed)... 2024-09-17T07:20:28.255 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Reconfiguring daemon node-exporter.a on smithi044 2024-09-17T07:20:28.255 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-17T07:20:28.255 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Reconfiguring daemon alertmanager.a on smithi044 2024-09-17T07:20:28.255 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Reconfiguring iscsi.foo.smithi044.elmtfk (dependencies changed)... 2024-09-17T07:20:28.255 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Reconfiguring daemon iscsi.foo.smithi044.elmtfk on smithi044 2024-09-17T07:20:28.255 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T07:20:28.255 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Reconfiguring daemon prometheus.a on smithi179 2024-09-17T07:20:28.255 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Reconfiguring node-exporter.b (dependencies changed)... 2024-09-17T07:20:28.255 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Reconfiguring daemon node-exporter.b on smithi179 2024-09-17T07:20:28.255 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Reconfiguring grafana.a (dependencies changed)... 2024-09-17T07:20:28.255 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-17T07:20:28.255 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Reconfiguring daemon grafana.a on smithi179 2024-09-17T07:20:28.256 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T07:20:28.256 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T07:20:28.256 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Upgrade: Updating mgr.y 2024-09-17T07:20:28.256 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Deploying daemon mgr.y on smithi044 2024-09-17T07:20:28.256 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:20:28.256 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T07:20:28.256 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T07:20:28.256 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Failing over to other MGR 2024-09-17T07:20:28.256 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: inventory: adjusted host smithi179 addr 'smithi179' -> '172.21.15.179' 2024-09-17T07:20:28.256 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: [17/Sep/2024:07:19:06] ENGINE Bus STARTING 2024-09-17T07:20:28.256 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: [17/Sep/2024:07:19:06] ENGINE Serving on https://172.21.15.44:7150 2024-09-17T07:20:28.256 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: [17/Sep/2024:07:19:06] ENGINE Error in HTTPServer.serve 2024-09-17T07:20:28.256 INFO:journalctl@ceph.mon.a.smithi044.stdout: Traceback (most recent call last): 2024-09-17T07:20:28.256 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T07:20:28.256 INFO:journalctl@ceph.mon.a.smithi044.stdout: self._connections.run(self.expiration_interval) 2024-09-17T07:20:28.257 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T07:20:28.257 INFO:journalctl@ceph.mon.a.smithi044.stdout: self._run(expiration_interval) 2024-09-17T07:20:28.257 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T07:20:28.257 INFO:journalctl@ceph.mon.a.smithi044.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T07:20:28.257 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T07:20:28.257 INFO:journalctl@ceph.mon.a.smithi044.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T07:20:28.257 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T07:20:28.257 INFO:journalctl@ceph.mon.a.smithi044.stdout: s = self.context.wrap_socket( 2024-09-17T07:20:28.257 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: mkfs 817ae996-74c3-11ef-bceb-c7b262605968 2024-09-17T07:20:28.258 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-09-17T07:20:28.258 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-09-17T07:20:28.258 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Reconfiguring node-exporter.a (dependencies changed)... 2024-09-17T07:20:28.258 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Reconfiguring daemon node-exporter.a on smithi044 2024-09-17T07:20:28.258 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-17T07:20:28.258 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Reconfiguring daemon alertmanager.a on smithi044 2024-09-17T07:20:28.258 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Reconfiguring iscsi.foo.smithi044.elmtfk (dependencies changed)... 2024-09-17T07:20:28.258 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Reconfiguring daemon iscsi.foo.smithi044.elmtfk on smithi044 2024-09-17T07:20:28.258 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T07:20:28.258 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Reconfiguring daemon prometheus.a on smithi179 2024-09-17T07:20:28.258 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Reconfiguring node-exporter.b (dependencies changed)... 2024-09-17T07:20:28.258 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Reconfiguring daemon node-exporter.b on smithi179 2024-09-17T07:20:28.258 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Reconfiguring grafana.a (dependencies changed)... 2024-09-17T07:20:28.258 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-17T07:20:28.259 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Reconfiguring daemon grafana.a on smithi179 2024-09-17T07:20:28.259 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T07:20:28.259 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T07:20:28.259 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Upgrade: Updating mgr.y 2024-09-17T07:20:28.259 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Deploying daemon mgr.y on smithi044 2024-09-17T07:20:28.259 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:20:28.259 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T07:20:28.259 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T07:20:28.259 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Failing over to other MGR 2024-09-17T07:20:28.259 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: inventory: adjusted host smithi179 addr 'smithi179' -> '172.21.15.179' 2024-09-17T07:20:28.259 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: [17/Sep/2024:07:19:06] ENGINE Bus STARTING 2024-09-17T07:20:28.259 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: [17/Sep/2024:07:19:06] ENGINE Serving on https://172.21.15.44:7150 2024-09-17T07:20:28.259 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: [17/Sep/2024:07:19:06] ENGINE Error in HTTPServer.serve 2024-09-17T07:20:28.260 INFO:journalctl@ceph.mon.c.smithi044.stdout: Traceback (most recent call last): 2024-09-17T07:20:28.260 INFO:journalctl@ceph.mon.c.smithi044.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T07:20:28.260 INFO:journalctl@ceph.mon.c.smithi044.stdout: self._connections.run(self.expiration_interval) 2024-09-17T07:20:28.260 INFO:journalctl@ceph.mon.c.smithi044.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T07:20:28.260 INFO:journalctl@ceph.mon.c.smithi044.stdout: self._run(expiration_interval) 2024-09-17T07:20:28.260 INFO:journalctl@ceph.mon.c.smithi044.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T07:20:28.260 INFO:journalctl@ceph.mon.c.smithi044.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T07:20:28.260 INFO:journalctl@ceph.mon.c.smithi044.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T07:20:28.260 INFO:journalctl@ceph.mon.c.smithi044.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T07:20:28.260 INFO:journalctl@ceph.mon.c.smithi044.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T07:20:28.260 INFO:journalctl@ceph.mon.c.smithi044.stdout: s = self.context.wrap_socket( 2024-09-17T07:20:28.261 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T07:20:28.261 INFO:journalctl@ceph.mon.a.smithi044.stdout: return self.sslsocket_class._create( 2024-09-17T07:20:28.261 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T07:20:28.261 INFO:journalctl@ceph.mon.a.smithi044.stdout: self.do_handshake() 2024-09-17T07:20:28.261 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T07:20:28.261 INFO:journalctl@ceph.mon.a.smithi044.stdout: self._sslobj.do_handshake() 2024-09-17T07:20:28.261 INFO:journalctl@ceph.mon.a.smithi044.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T07:20:28.261 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: [17/Sep/2024:07:19:06] ENGINE Serving on http://172.21.15.44:8765 2024-09-17T07:20:28.261 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: [17/Sep/2024:07:19:06] ENGINE Bus STARTED 2024-09-17T07:20:28.261 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:20:28.261 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Reconfiguring iscsi.foo.smithi044.elmtfk (dependencies changed)... 2024-09-17T07:20:28.261 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Reconfiguring daemon iscsi.foo.smithi044.elmtfk on smithi044 2024-09-17T07:20:28.261 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T07:20:28.261 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Reconfiguring daemon prometheus.a on smithi179 2024-09-17T07:20:28.262 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Upgrade: Updating mgr.x 2024-09-17T07:20:28.262 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Deploying daemon mgr.x on smithi179 2024-09-17T07:20:28.262 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:20:28.262 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Upgrade: Setting container_image for all mgr 2024-09-17T07:20:28.262 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Upgrade: It appears safe to stop mon.a 2024-09-17T07:20:28.262 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Upgrade: Updating mon.a 2024-09-17T07:20:28.262 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Deploying daemon mon.a on smithi044 2024-09-17T07:20:28.262 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-09-17T07:20:28.262 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T07:20:28.262 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: daemon iscsi.foo.smithi044.elmtfk on smithi044 is in unknown state 2024-09-17T07:20:28.262 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: daemon prometheus.a on smithi179 is in unknown state 2024-09-17T07:20:28.262 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: mgrmap e25: y(active, since 43s), standbys: x 2024-09-17T07:20:28.262 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: mgrmap e26: y(active, since 44s), standbys: x 2024-09-17T07:20:28.263 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Standby manager daemon x restarted 2024-09-17T07:20:28.263 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Standby manager daemon x started 2024-09-17T07:20:28.263 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: mgrmap e27: y(active, since 47s), standbys: x 2024-09-17T07:20:28.263 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Active manager daemon y restarted 2024-09-17T07:20:28.263 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Activating manager daemon y 2024-09-17T07:20:28.263 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: osdmap e76: 8 total, 8 up, 8 in 2024-09-17T07:20:28.263 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: mgrmap e28: y(active, starting, since 0.0127954s), standbys: x 2024-09-17T07:20:28.263 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Manager daemon y is now available 2024-09-17T07:20:28.263 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:20:28.263 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:20:28.263 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: mgrmap e29: y(active, since 1.01164s), standbys: x 2024-09-17T07:20:28.263 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: [17/Sep/2024:07:19:51] ENGINE Bus STARTING 2024-09-17T07:20:28.264 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: [17/Sep/2024:07:19:51] ENGINE Serving on https://172.21.15.44:7150 2024-09-17T07:20:28.264 INFO:journalctl@ceph.mon.c.smithi044.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T07:20:28.264 INFO:journalctl@ceph.mon.c.smithi044.stdout: return self.sslsocket_class._create( 2024-09-17T07:20:28.264 INFO:journalctl@ceph.mon.c.smithi044.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T07:20:28.264 INFO:journalctl@ceph.mon.c.smithi044.stdout: self.do_handshake() 2024-09-17T07:20:28.264 INFO:journalctl@ceph.mon.c.smithi044.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T07:20:28.264 INFO:journalctl@ceph.mon.c.smithi044.stdout: self._sslobj.do_handshake() 2024-09-17T07:20:28.264 INFO:journalctl@ceph.mon.c.smithi044.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T07:20:28.264 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: [17/Sep/2024:07:19:06] ENGINE Serving on http://172.21.15.44:8765 2024-09-17T07:20:28.264 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: [17/Sep/2024:07:19:06] ENGINE Bus STARTED 2024-09-17T07:20:28.265 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:20:28.265 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Reconfiguring iscsi.foo.smithi044.elmtfk (dependencies changed)... 2024-09-17T07:20:28.265 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Reconfiguring daemon iscsi.foo.smithi044.elmtfk on smithi044 2024-09-17T07:20:28.265 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T07:20:28.265 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Reconfiguring daemon prometheus.a on smithi179 2024-09-17T07:20:28.265 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Upgrade: Updating mgr.x 2024-09-17T07:20:28.265 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Deploying daemon mgr.x on smithi179 2024-09-17T07:20:28.265 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:20:28.265 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Upgrade: Setting container_image for all mgr 2024-09-17T07:20:28.265 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Upgrade: It appears safe to stop mon.a 2024-09-17T07:20:28.265 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Upgrade: Updating mon.a 2024-09-17T07:20:28.265 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Deploying daemon mon.a on smithi044 2024-09-17T07:20:28.265 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-09-17T07:20:28.266 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T07:20:28.266 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: daemon iscsi.foo.smithi044.elmtfk on smithi044 is in unknown state 2024-09-17T07:20:28.266 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: daemon prometheus.a on smithi179 is in unknown state 2024-09-17T07:20:28.266 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: mgrmap e25: y(active, since 43s), standbys: x 2024-09-17T07:20:28.266 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: mgrmap e26: y(active, since 44s), standbys: x 2024-09-17T07:20:28.266 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Standby manager daemon x restarted 2024-09-17T07:20:28.266 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Standby manager daemon x started 2024-09-17T07:20:28.266 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: mgrmap e27: y(active, since 47s), standbys: x 2024-09-17T07:20:28.266 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Active manager daemon y restarted 2024-09-17T07:20:28.266 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Activating manager daemon y 2024-09-17T07:20:28.266 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: osdmap e76: 8 total, 8 up, 8 in 2024-09-17T07:20:28.266 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: mgrmap e28: y(active, starting, since 0.0127954s), standbys: x 2024-09-17T07:20:28.266 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Manager daemon y is now available 2024-09-17T07:20:28.266 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T07:20:28.267 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T07:20:28.267 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: mgrmap e29: y(active, since 1.01164s), standbys: x 2024-09-17T07:20:28.267 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: [17/Sep/2024:07:19:51] ENGINE Bus STARTING 2024-09-17T07:20:28.267 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: [17/Sep/2024:07:19:51] ENGINE Serving on https://172.21.15.44:7150 2024-09-17T07:20:28.267 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: [17/Sep/2024:07:19:51] ENGINE Error in HTTPServer.serve 2024-09-17T07:20:28.267 INFO:journalctl@ceph.mon.a.smithi044.stdout: Traceback (most recent call last): 2024-09-17T07:20:28.267 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T07:20:28.267 INFO:journalctl@ceph.mon.a.smithi044.stdout: self._connections.run(self.expiration_interval) 2024-09-17T07:20:28.267 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T07:20:28.267 INFO:journalctl@ceph.mon.a.smithi044.stdout: self._run(expiration_interval) 2024-09-17T07:20:28.267 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T07:20:28.268 INFO:journalctl@ceph.mon.a.smithi044.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T07:20:28.268 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T07:20:28.268 INFO:journalctl@ceph.mon.a.smithi044.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T07:20:28.268 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T07:20:28.268 INFO:journalctl@ceph.mon.a.smithi044.stdout: s = self.context.wrap_socket( 2024-09-17T07:20:28.268 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T07:20:28.268 INFO:journalctl@ceph.mon.a.smithi044.stdout: return self.sslsocket_class._create( 2024-09-17T07:20:28.268 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T07:20:28.268 INFO:journalctl@ceph.mon.a.smithi044.stdout: self.do_handshake() 2024-09-17T07:20:28.268 INFO:journalctl@ceph.mon.a.smithi044.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T07:20:28.268 INFO:journalctl@ceph.mon.a.smithi044.stdout: self._sslobj.do_handshake() 2024-09-17T07:20:28.268 INFO:journalctl@ceph.mon.a.smithi044.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T07:20:28.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2024-09-17T07:20:28.268 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Cluster is now healthy 2024-09-17T07:20:28.269 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: pgmap v4: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:20:28.269 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: mgrmap e30: y(active, since 2s), standbys: x 2024-09-17T07:20:28.269 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: [17/Sep/2024:07:19:52] ENGINE Serving on http://172.21.15.44:8765 2024-09-17T07:20:28.269 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: [17/Sep/2024:07:19:52] ENGINE Bus STARTED 2024-09-17T07:20:28.269 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/2556707789' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:20:28.269 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='client.25351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:20:28.269 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='client.44190 172.21.15.44:0/2380431437' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:20:28.269 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: pgmap v5: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:20:28.269 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: mgrmap e31: y(active, since 4s), standbys: x 2024-09-17T07:20:28.269 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.269 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.269 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.269 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.269 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.269 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.270 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.270 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.270 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: pgmap v6: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:20:28.270 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: [17/Sep/2024:07:19:51] ENGINE Error in HTTPServer.serve 2024-09-17T07:20:28.270 INFO:journalctl@ceph.mon.c.smithi044.stdout: Traceback (most recent call last): 2024-09-17T07:20:28.270 INFO:journalctl@ceph.mon.c.smithi044.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T07:20:28.270 INFO:journalctl@ceph.mon.c.smithi044.stdout: self._connections.run(self.expiration_interval) 2024-09-17T07:20:28.270 INFO:journalctl@ceph.mon.c.smithi044.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T07:20:28.270 INFO:journalctl@ceph.mon.c.smithi044.stdout: self._run(expiration_interval) 2024-09-17T07:20:28.270 INFO:journalctl@ceph.mon.c.smithi044.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T07:20:28.271 INFO:journalctl@ceph.mon.c.smithi044.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T07:20:28.271 INFO:journalctl@ceph.mon.c.smithi044.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T07:20:28.271 INFO:journalctl@ceph.mon.c.smithi044.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T07:20:28.271 INFO:journalctl@ceph.mon.c.smithi044.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T07:20:28.271 INFO:journalctl@ceph.mon.c.smithi044.stdout: s = self.context.wrap_socket( 2024-09-17T07:20:28.271 INFO:journalctl@ceph.mon.c.smithi044.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T07:20:28.271 INFO:journalctl@ceph.mon.c.smithi044.stdout: return self.sslsocket_class._create( 2024-09-17T07:20:28.271 INFO:journalctl@ceph.mon.c.smithi044.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T07:20:28.271 INFO:journalctl@ceph.mon.c.smithi044.stdout: self.do_handshake() 2024-09-17T07:20:28.271 INFO:journalctl@ceph.mon.c.smithi044.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T07:20:28.271 INFO:journalctl@ceph.mon.c.smithi044.stdout: self._sslobj.do_handshake() 2024-09-17T07:20:28.271 INFO:journalctl@ceph.mon.c.smithi044.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T07:20:28.271 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2024-09-17T07:20:28.271 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Cluster is now healthy 2024-09-17T07:20:28.271 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: pgmap v4: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:20:28.272 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: mgrmap e30: y(active, since 2s), standbys: x 2024-09-17T07:20:28.272 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: [17/Sep/2024:07:19:52] ENGINE Serving on http://172.21.15.44:8765 2024-09-17T07:20:28.272 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: [17/Sep/2024:07:19:52] ENGINE Bus STARTED 2024-09-17T07:20:28.272 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/2556707789' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:20:28.272 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='client.25351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:20:28.272 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='client.44190 172.21.15.44:0/2380431437' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:20:28.272 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: pgmap v5: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:20:28.272 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: mgrmap e31: y(active, since 4s), standbys: x 2024-09-17T07:20:28.272 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.272 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.272 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.272 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.272 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.272 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.273 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.273 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.273 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: pgmap v6: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:20:28.273 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: pgmap v7: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-17T07:20:28.273 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/3155806008' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:20:28.273 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.273 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: pgmap v8: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-17T07:20:28.273 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.274 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:28.274 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:20:28.274 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:20:28.274 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-17T07:20:28.274 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.274 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: overall HEALTH_OK 2024-09-17T07:20:28.274 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:20:28.274 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:20:28.274 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T07:20:28.274 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-17T07:20:28.274 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Upgrade: It appears safe to stop mon.c 2024-09-17T07:20:28.274 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/1958833622' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2084457509"}]: access denied 2024-09-17T07:20:28.274 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T07:20:28.274 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T07:20:28.274 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.275 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T07:20:28.275 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T07:20:28.275 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:28.275 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T07:20:28.275 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T07:20:28.275 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: mon.c calling monitor election 2024-09-17T07:20:28.275 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: mon.a calling monitor election 2024-09-17T07:20:28.275 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T07:20:28.275 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: monmap e3: 3 mons at {a=[v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0],b=[v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0],c=[v2:172.21.15.44:3301/0,v1:172.21.15.44:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T07:20:28.275 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: fsmap 2024-09-17T07:20:28.275 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: osdmap e76: 8 total, 8 up, 8 in 2024-09-17T07:20:28.275 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: mgrmap e31: y(active, since 17s), standbys: x 2024-09-17T07:20:28.275 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-09-17T07:20:28.276 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T07:20:28.276 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: daemon iscsi.foo.smithi044.elmtfk on smithi044 is in unknown state 2024-09-17T07:20:28.276 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: daemon prometheus.a on smithi179 is in unknown state 2024-09-17T07:20:28.276 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.276 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.276 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:20:28.276 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-09-17T07:20:28.276 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-09-17T07:20:28.276 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:20:28.276 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/2774548008' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:20:28.276 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='client.44238 172.21.15.44:0/2800210551' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:20:28.276 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:20:28.276 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.277 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.277 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:20:28.277 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.277 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.277 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: pgmap v18: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:20:28.277 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.277 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.277 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:28.277 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:20:28.277 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:20:28.277 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: pgmap v19: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2024-09-17T07:20:28.277 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: pgmap v20: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2024-09-17T07:20:28.277 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.277 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:20:28.278 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: pgmap v7: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-17T07:20:28.278 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/3155806008' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:20:28.278 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.278 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: pgmap v8: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-17T07:20:28.278 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.278 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:28.278 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:20:28.279 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:20:28.279 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-17T07:20:28.279 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.279 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: overall HEALTH_OK 2024-09-17T07:20:28.279 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:20:28.279 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:20:28.279 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T07:20:28.279 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-17T07:20:28.279 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Upgrade: It appears safe to stop mon.c 2024-09-17T07:20:28.279 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/1958833622' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2084457509"}]: access denied 2024-09-17T07:20:28.279 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T07:20:28.279 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T07:20:28.279 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.279 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T07:20:28.280 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T07:20:28.280 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:28.280 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T07:20:28.280 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T07:20:28.280 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: mon.c calling monitor election 2024-09-17T07:20:28.280 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: mon.a calling monitor election 2024-09-17T07:20:28.280 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T07:20:28.280 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:20:28.280 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T07:20:28.280 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-09-17T07:20:28.281 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Upgrade: It appears safe to stop mon.b 2024-09-17T07:20:28.281 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: pgmap v21: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2024-09-17T07:20:28.281 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.281 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T07:20:28.281 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T07:20:28.281 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:28.281 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:20:28.281 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: mon.a calling monitor election 2024-09-17T07:20:28.281 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: mon.b calling monitor election 2024-09-17T07:20:28.281 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: mon.c calling monitor election 2024-09-17T07:20:28.281 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:20:28.281 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:20:28.282 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T07:20:28.282 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: monmap e4: 3 mons at {a=[v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0],b=[v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0],c=[v2:172.21.15.44:3301/0,v1:172.21.15.44:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T07:20:28.282 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: fsmap 2024-09-17T07:20:28.282 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: osdmap e76: 8 total, 8 up, 8 in 2024-09-17T07:20:28.282 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: mgrmap e31: y(active, since 36s), standbys: x 2024-09-17T07:20:28.282 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-09-17T07:20:28.282 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T07:20:28.282 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: daemon iscsi.foo.smithi044.elmtfk on smithi044 is in unknown state 2024-09-17T07:20:28.282 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: daemon prometheus.a on smithi179 is in unknown state 2024-09-17T07:20:28.282 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.282 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.282 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:20:28.282 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/883135932' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:20:28.283 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: monmap e3: 3 mons at {a=[v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0],b=[v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0],c=[v2:172.21.15.44:3301/0,v1:172.21.15.44:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T07:20:28.283 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: fsmap 2024-09-17T07:20:28.283 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: osdmap e76: 8 total, 8 up, 8 in 2024-09-17T07:20:28.283 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: mgrmap e31: y(active, since 17s), standbys: x 2024-09-17T07:20:28.283 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-09-17T07:20:28.283 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T07:20:28.283 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: daemon iscsi.foo.smithi044.elmtfk on smithi044 is in unknown state 2024-09-17T07:20:28.283 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: daemon prometheus.a on smithi179 is in unknown state 2024-09-17T07:20:28.283 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.284 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.284 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:20:28.284 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-09-17T07:20:28.284 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-09-17T07:20:28.284 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:20:28.284 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/2774548008' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:20:28.284 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='client.44238 172.21.15.44:0/2800210551' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:20:28.284 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:20:28.284 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.284 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.284 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:20:28.284 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.284 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.285 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: pgmap v18: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:20:28.285 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.285 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.285 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:28.285 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:20:28.285 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:20:28.285 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: pgmap v19: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 86 B/s rd, 0 op/s 2024-09-17T07:20:28.285 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: pgmap v20: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2024-09-17T07:20:28.285 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.285 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:20:28.285 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:20:28.285 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T07:20:28.285 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-09-17T07:20:28.285 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Upgrade: It appears safe to stop mon.b 2024-09-17T07:20:28.286 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: pgmap v21: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 104 B/s rd, 0 op/s 2024-09-17T07:20:28.286 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.286 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T07:20:28.286 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T07:20:28.286 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:28.286 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T07:20:28.286 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: mon.a calling monitor election 2024-09-17T07:20:28.286 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: mon.b calling monitor election 2024-09-17T07:20:28.286 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: mon.c calling monitor election 2024-09-17T07:20:28.286 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T07:20:28.286 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T07:20:28.286 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T07:20:28.286 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: monmap e4: 3 mons at {a=[v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0],b=[v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0],c=[v2:172.21.15.44:3301/0,v1:172.21.15.44:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T07:20:28.286 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: fsmap 2024-09-17T07:20:28.287 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: osdmap e76: 8 total, 8 up, 8 in 2024-09-17T07:20:28.287 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: mgrmap e31: y(active, since 36s), standbys: x 2024-09-17T07:20:28.287 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-09-17T07:20:28.287 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T07:20:28.287 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: daemon iscsi.foo.smithi044.elmtfk on smithi044 is in unknown state 2024-09-17T07:20:28.287 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: daemon prometheus.a on smithi179 is in unknown state 2024-09-17T07:20:28.287 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.287 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:28.287 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:20:28.287 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:27 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/883135932' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:20:29.113 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:28 smithi179 ceph-mon[50085]: pgmap v24: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:20:29.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:28 smithi179 ceph-mon[50085]: from='client.54181 172.21.15.44:0/1144955776' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:20:29.245 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:28 smithi044 ceph-mon[58191]: pgmap v24: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:20:29.245 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:28 smithi044 ceph-mon[58191]: from='client.54181 172.21.15.44:0/1144955776' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:20:29.245 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:28 smithi044 ceph-mon[60356]: pgmap v24: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:20:29.245 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:28 smithi044 ceph-mon[60356]: from='client.54181 172.21.15.44:0/1144955776' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:20:30.824 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:20:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:20:30.381+0000 7f5a7ae47640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-17T07:20:31.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:30 smithi179 ceph-mon[50085]: pgmap v25: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:20:31.114 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:30 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Scheduled restart job, restart counter is at 6. 2024-09-17T07:20:31.114 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:30 smithi179 systemd[1]: Stopped Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:20:31.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:30 smithi179 systemd[1]: Starting Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:20:31.324 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:30 smithi044 ceph-mon[60356]: pgmap v25: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:20:31.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:30 smithi044 ceph-mon[58191]: pgmap v25: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:20:31.633 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 podman[50742]: 2024-09-17 07:20:31.279841381 +0000 UTC m=+0.017073675 image pull de242295e2257c37c8cadfd962369228f8f10b2d48a44259b65fef44ad4f6490 docker.io/prom/prometheus:v2.18.1 2024-09-17T07:20:31.633 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 podman[50742]: 2024-09-17 07:20:31.423129235 +0000 UTC m=+0.160361529 container create f81821581999dc2a26d177b84452c67065f6393ff857984bdb2fd073050aca1b (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:20:31.633 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 podman[50742]: 2024-09-17 07:20:31.633116788 +0000 UTC m=+0.370349088 container init f81821581999dc2a26d177b84452c67065f6393ff857984bdb2fd073050aca1b (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:20:32.011 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 podman[50742]: 2024-09-17 07:20:31.636129296 +0000 UTC m=+0.373361590 container start f81821581999dc2a26d177b84452c67065f6393ff857984bdb2fd073050aca1b (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:20:32.011 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.666Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-17T07:20:32.011 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.666Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2024-09-17T07:20:32.012 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.666Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2024-09-17T07:20:32.012 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.666Z caller=main.go:339 host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi179 (none))" 2024-09-17T07:20:32.012 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.666Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T07:20:32.012 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.666Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T07:20:32.012 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.668Z caller=main.go:678 msg="Starting TSDB ..." 2024-09-17T07:20:32.012 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.668Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2024-09-17T07:20:32.012 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.670Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2024-09-17T07:20:32.012 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.670Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=13 2024-09-17T07:20:32.012 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.677Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=13 2024-09-17T07:20:32.012 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.678Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=13 2024-09-17T07:20:32.012 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.678Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=13 2024-09-17T07:20:32.012 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.678Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=13 2024-09-17T07:20:32.012 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.678Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=13 2024-09-17T07:20:32.013 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.678Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=13 2024-09-17T07:20:32.013 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.678Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=7 maxSegment=13 2024-09-17T07:20:32.013 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.678Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=8 maxSegment=13 2024-09-17T07:20:32.013 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.679Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=9 maxSegment=13 2024-09-17T07:20:32.013 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.679Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=10 maxSegment=13 2024-09-17T07:20:32.013 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.679Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=11 maxSegment=13 2024-09-17T07:20:32.013 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.680Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=12 maxSegment=13 2024-09-17T07:20:32.014 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.680Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=13 maxSegment=13 2024-09-17T07:20:32.014 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.680Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=9.788652ms 2024-09-17T07:20:32.014 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.682Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2024-09-17T07:20:32.014 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.682Z caller=main.go:695 msg="TSDB started" 2024-09-17T07:20:32.014 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.682Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T07:20:32.014 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.683Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2024-09-17T07:20:32.014 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.683Z caller=main.go:561 msg="Stopping notify discovery manager..." 2024-09-17T07:20:32.014 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.683Z caller=main.go:583 msg="Stopping scrape manager..." 2024-09-17T07:20:32.014 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.683Z caller=main.go:557 msg="Notify discovery manager stopped" 2024-09-17T07:20:32.014 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.683Z caller=main.go:543 msg="Scrape discovery manager stopped" 2024-09-17T07:20:32.014 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.683Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2024-09-17T07:20:32.014 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.683Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2024-09-17T07:20:32.014 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.683Z caller=main.go:577 msg="Scrape manager stopped" 2024-09-17T07:20:32.014 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.683Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2024-09-17T07:20:32.015 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=info ts=2024-09-17T07:20:31.683Z caller=main.go:749 msg="Notifier manager stopped" 2024-09-17T07:20:32.015 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[50770]: level=error ts=2024-09-17T07:20:31.683Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 14: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain\n line 31: field http_sd_configs not found in type config.plain\n line 45: field http_sd_configs not found in type config.plain\n line 49: field http_sd_configs not found in type config.plain" 2024-09-17T07:20:32.015 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 bash[50742]: f81821581999dc2a26d177b84452c67065f6393ff857984bdb2fd073050aca1b 2024-09-17T07:20:32.015 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 systemd[1]: Started Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:20:32.015 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:31 smithi179 podman[50806]: 2024-09-17 07:20:31.741086917 +0000 UTC m=+0.050375053 container died f81821581999dc2a26d177b84452c67065f6393ff857984bdb2fd073050aca1b (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:20:32.613 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:32 smithi179 podman[50806]: 2024-09-17 07:20:32.345858845 +0000 UTC m=+0.655147017 container remove f81821581999dc2a26d177b84452c67065f6393ff857984bdb2fd073050aca1b (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:20:32.614 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:32 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2024-09-17T07:20:32.614 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:32 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Failed with result 'exit-code'. 2024-09-17T07:20:33.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:32 smithi044 ceph-mon[58191]: pgmap v26: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:20:33.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:32 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:33.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:32 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:33.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:32 smithi044 ceph-mon[60356]: pgmap v26: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:20:33.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:32 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:33.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:32 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:33.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:32 smithi179 ceph-mon[50085]: pgmap v26: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:20:33.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:32 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:33.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:32 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:34.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:34 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:34.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:34 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:34.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:34 smithi044 ceph-mon[58191]: pgmap v27: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:20:34.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:34 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:34.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:34 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:34.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:34 smithi044 ceph-mon[60356]: pgmap v27: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:20:34.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:34 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:34.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:34 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:34.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:34 smithi179 ceph-mon[50085]: pgmap v27: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:20:37.003 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:36 smithi044 ceph-mon[58191]: pgmap v28: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:20:37.003 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:36 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:37.003 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:36 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:37.003 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:36 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:37.003 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:36 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:20:37.004 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:36 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:37.004 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:36 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T07:20:37.004 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:36 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T07:20:37.004 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:36 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:37.004 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:36 smithi044 ceph-mon[60356]: pgmap v28: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:20:37.004 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:36 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:37.004 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:36 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:37.004 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:36 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:37.004 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:36 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:20:37.004 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:36 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:37.005 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:36 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T07:20:37.005 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:36 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T07:20:37.005 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:36 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:37.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:36 smithi179 ceph-mon[50085]: pgmap v28: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:20:37.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:36 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:37.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:36 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:37.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:36 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:37.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:36 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:20:37.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:36 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:37.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:36 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T07:20:37.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:36 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T07:20:37.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:36 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:38.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:37 smithi044 ceph-mon[58191]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:20:38.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:37 smithi044 ceph-mon[58191]: pgmap v29: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 97 B/s rd, 0 op/s 2024-09-17T07:20:38.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:37 smithi044 ceph-mon[58191]: pgmap v30: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 120 B/s rd, 0 op/s 2024-09-17T07:20:38.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:37 smithi044 ceph-mon[58191]: Reconfiguring mon.a (monmap changed)... 2024-09-17T07:20:38.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:37 smithi044 ceph-mon[58191]: Reconfiguring daemon mon.a on smithi044 2024-09-17T07:20:38.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:37 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:38.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:37 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:38.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:37 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T07:20:38.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:37 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T07:20:38.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:37 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:38.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:37 smithi044 ceph-mon[60356]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:20:38.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:37 smithi044 ceph-mon[60356]: pgmap v29: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 97 B/s rd, 0 op/s 2024-09-17T07:20:38.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:37 smithi044 ceph-mon[60356]: pgmap v30: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 120 B/s rd, 0 op/s 2024-09-17T07:20:38.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:37 smithi044 ceph-mon[60356]: Reconfiguring mon.a (monmap changed)... 2024-09-17T07:20:38.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:37 smithi044 ceph-mon[60356]: Reconfiguring daemon mon.a on smithi044 2024-09-17T07:20:38.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:37 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:38.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:37 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:38.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:37 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T07:20:38.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:37 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T07:20:38.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:37 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:38.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:37 smithi179 ceph-mon[50085]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:20:38.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:37 smithi179 ceph-mon[50085]: pgmap v29: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 97 B/s rd, 0 op/s 2024-09-17T07:20:38.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:37 smithi179 ceph-mon[50085]: pgmap v30: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 120 B/s rd, 0 op/s 2024-09-17T07:20:38.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:37 smithi179 ceph-mon[50085]: Reconfiguring mon.a (monmap changed)... 2024-09-17T07:20:38.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:37 smithi179 ceph-mon[50085]: Reconfiguring daemon mon.a on smithi044 2024-09-17T07:20:38.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:37 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:38.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:37 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:38.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:37 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T07:20:38.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:37 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T07:20:38.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:37 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:39.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:38 smithi044 ceph-mon[58191]: Reconfiguring mgr.y (monmap changed)... 2024-09-17T07:20:39.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:38 smithi044 ceph-mon[58191]: Reconfiguring daemon mgr.y on smithi044 2024-09-17T07:20:39.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:38 smithi044 ceph-mon[60356]: Reconfiguring mgr.y (monmap changed)... 2024-09-17T07:20:39.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:38 smithi044 ceph-mon[60356]: Reconfiguring daemon mgr.y on smithi044 2024-09-17T07:20:39.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:38 smithi179 ceph-mon[50085]: Reconfiguring mgr.y (monmap changed)... 2024-09-17T07:20:39.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:38 smithi179 ceph-mon[50085]: Reconfiguring daemon mgr.y on smithi044 2024-09-17T07:20:40.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:39 smithi044 ceph-mon[58191]: pgmap v31: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 120 B/s rd, 0 op/s 2024-09-17T07:20:40.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:40.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:40.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T07:20:40.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T07:20:40.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:40.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:39 smithi044 ceph-mon[60356]: pgmap v31: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 120 B/s rd, 0 op/s 2024-09-17T07:20:40.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:40.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:40.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T07:20:40.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T07:20:40.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:40.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:39 smithi179 ceph-mon[50085]: pgmap v31: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 120 B/s rd, 0 op/s 2024-09-17T07:20:40.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:40.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:40.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T07:20:40.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T07:20:40.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:41.025 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:40 smithi044 ceph-mon[60356]: Reconfiguring mon.c (monmap changed)... 2024-09-17T07:20:41.025 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:40 smithi044 ceph-mon[60356]: Reconfiguring daemon mon.c on smithi044 2024-09-17T07:20:41.025 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:40 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:41.025 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:40 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:41.025 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:40 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-17T07:20:41.026 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:40 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:41.026 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:40 smithi044 ceph-mon[58191]: Reconfiguring mon.c (monmap changed)... 2024-09-17T07:20:41.026 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:40 smithi044 ceph-mon[58191]: Reconfiguring daemon mon.c on smithi044 2024-09-17T07:20:41.026 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:40 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:41.026 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:40 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:41.026 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:40 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-17T07:20:41.026 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:40 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:41.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:40 smithi179 ceph-mon[50085]: Reconfiguring mon.c (monmap changed)... 2024-09-17T07:20:41.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:40 smithi179 ceph-mon[50085]: Reconfiguring daemon mon.c on smithi044 2024-09-17T07:20:41.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:40 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:41.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:40 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:41.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:40 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-17T07:20:41.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:40 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:42.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:41 smithi044 ceph-mon[58191]: pgmap v32: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:20:42.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:41 smithi044 ceph-mon[58191]: Reconfiguring osd.0 (monmap changed)... 2024-09-17T07:20:42.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:41 smithi044 ceph-mon[58191]: Reconfiguring daemon osd.0 on smithi044 2024-09-17T07:20:42.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:41 smithi044 ceph-mon[60356]: pgmap v32: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:20:42.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:41 smithi044 ceph-mon[60356]: Reconfiguring osd.0 (monmap changed)... 2024-09-17T07:20:42.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:41 smithi044 ceph-mon[60356]: Reconfiguring daemon osd.0 on smithi044 2024-09-17T07:20:42.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:41 smithi179 ceph-mon[50085]: pgmap v32: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:20:42.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:41 smithi179 ceph-mon[50085]: Reconfiguring osd.0 (monmap changed)... 2024-09-17T07:20:42.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:41 smithi179 ceph-mon[50085]: Reconfiguring daemon osd.0 on smithi044 2024-09-17T07:20:42.670 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Scheduled restart job, restart counter is at 7. 2024-09-17T07:20:42.670 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 systemd[1]: Stopped Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:20:42.670 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 systemd[1]: Starting Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:20:42.670 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 podman[51320]: 2024-09-17 07:20:42.55375206 +0000 UTC m=+0.114067935 container create 86afd61260cb79c6b6d2bb18f5a492a59d96de49e79c6ddadde1596b37cdbabd (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:20:42.670 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 podman[51320]: 2024-09-17 07:20:42.457931245 +0000 UTC m=+0.018247099 image pull de242295e2257c37c8cadfd962369228f8f10b2d48a44259b65fef44ad4f6490 docker.io/prom/prometheus:v2.18.1 2024-09-17T07:20:43.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-mon[50085]: from='client.54199 172.21.15.44:0/3095273741' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:20:43.114 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 podman[51320]: 2024-09-17 07:20:42.78936641 +0000 UTC m=+0.349682265 container init 86afd61260cb79c6b6d2bb18f5a492a59d96de49e79c6ddadde1596b37cdbabd (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:20:43.114 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 podman[51320]: 2024-09-17 07:20:42.792429658 +0000 UTC m=+0.352745492 container start 86afd61260cb79c6b6d2bb18f5a492a59d96de49e79c6ddadde1596b37cdbabd (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:20:43.114 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.823Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-17T07:20:43.114 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.823Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2024-09-17T07:20:43.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.823Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2024-09-17T07:20:43.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.823Z caller=main.go:339 host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi179 (none))" 2024-09-17T07:20:43.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.823Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T07:20:43.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.823Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T07:20:43.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.824Z caller=main.go:678 msg="Starting TSDB ..." 2024-09-17T07:20:43.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.824Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2024-09-17T07:20:43.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.826Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2024-09-17T07:20:43.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.827Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=14 2024-09-17T07:20:43.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.833Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=14 2024-09-17T07:20:43.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.833Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=14 2024-09-17T07:20:43.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.833Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=14 2024-09-17T07:20:43.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.833Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=14 2024-09-17T07:20:43.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.834Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=14 2024-09-17T07:20:43.116 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.834Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=14 2024-09-17T07:20:43.116 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.834Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=7 maxSegment=14 2024-09-17T07:20:43.117 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.834Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=8 maxSegment=14 2024-09-17T07:20:43.117 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.834Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=9 maxSegment=14 2024-09-17T07:20:43.117 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.835Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=10 maxSegment=14 2024-09-17T07:20:43.117 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.835Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=11 maxSegment=14 2024-09-17T07:20:43.117 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.836Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=12 maxSegment=14 2024-09-17T07:20:43.117 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.836Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=13 maxSegment=14 2024-09-17T07:20:43.120 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.836Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=14 maxSegment=14 2024-09-17T07:20:43.120 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.836Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=9.900014ms 2024-09-17T07:20:43.120 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.838Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2024-09-17T07:20:43.121 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.838Z caller=main.go:695 msg="TSDB started" 2024-09-17T07:20:43.121 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.838Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T07:20:43.121 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.839Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2024-09-17T07:20:43.121 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.839Z caller=main.go:561 msg="Stopping notify discovery manager..." 2024-09-17T07:20:43.121 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.839Z caller=main.go:583 msg="Stopping scrape manager..." 2024-09-17T07:20:43.121 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.839Z caller=main.go:557 msg="Notify discovery manager stopped" 2024-09-17T07:20:43.121 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.839Z caller=main.go:543 msg="Scrape discovery manager stopped" 2024-09-17T07:20:43.122 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.839Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2024-09-17T07:20:43.122 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.839Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2024-09-17T07:20:43.122 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.839Z caller=main.go:577 msg="Scrape manager stopped" 2024-09-17T07:20:43.122 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.839Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2024-09-17T07:20:43.122 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=info ts=2024-09-17T07:20:42.839Z caller=main.go:749 msg="Notifier manager stopped" 2024-09-17T07:20:43.122 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51335]: level=error ts=2024-09-17T07:20:42.839Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 14: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain\n line 31: field http_sd_configs not found in type config.plain\n line 45: field http_sd_configs not found in type config.plain\n line 49: field http_sd_configs not found in type config.plain" 2024-09-17T07:20:43.123 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 bash[51320]: 86afd61260cb79c6b6d2bb18f5a492a59d96de49e79c6ddadde1596b37cdbabd 2024-09-17T07:20:43.123 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 systemd[1]: Started Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:20:43.123 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:42 smithi179 podman[51352]: 2024-09-17 07:20:42.882638835 +0000 UTC m=+0.035085533 container died 86afd61260cb79c6b6d2bb18f5a492a59d96de49e79c6ddadde1596b37cdbabd (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:20:43.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:42 smithi044 ceph-mon[58191]: from='client.54199 172.21.15.44:0/3095273741' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:20:43.225 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:42 smithi044 ceph-mon[60356]: from='client.54199 172.21.15.44:0/3095273741' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:20:43.864 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:43 smithi179 podman[51352]: 2024-09-17 07:20:43.443070735 +0000 UTC m=+0.595517433 container remove 86afd61260cb79c6b6d2bb18f5a492a59d96de49e79c6ddadde1596b37cdbabd (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:20:43.864 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:43 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2024-09-17T07:20:43.864 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:43 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Failed with result 'exit-code'. 2024-09-17T07:20:44.024 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:43 smithi044 ceph-mon[60356]: pgmap v33: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:20:44.024 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:43 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:44.024 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:43 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:44.025 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:43 smithi044 ceph-mon[60356]: Reconfiguring osd.1 (monmap changed)... 2024-09-17T07:20:44.025 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:43 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-17T07:20:44.025 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:43 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:44.025 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:43 smithi044 ceph-mon[60356]: Reconfiguring daemon osd.1 on smithi044 2024-09-17T07:20:44.025 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:43 smithi044 ceph-mon[60356]: from='client.44322 172.21.15.44:0/3395778652' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/1355368272"}]: access denied 2024-09-17T07:20:44.025 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:43 smithi044 ceph-mon[58191]: pgmap v33: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:20:44.025 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:43 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:44.025 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:43 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:44.025 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:43 smithi044 ceph-mon[58191]: Reconfiguring osd.1 (monmap changed)... 2024-09-17T07:20:44.025 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:43 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-17T07:20:44.025 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:43 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:44.026 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:43 smithi044 ceph-mon[58191]: Reconfiguring daemon osd.1 on smithi044 2024-09-17T07:20:44.026 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:43 smithi044 ceph-mon[58191]: from='client.44322 172.21.15.44:0/3395778652' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/1355368272"}]: access denied 2024-09-17T07:20:44.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:43 smithi179 ceph-mon[50085]: pgmap v33: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:20:44.368 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:43 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:44.369 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:43 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:44.369 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:43 smithi179 ceph-mon[50085]: Reconfiguring osd.1 (monmap changed)... 2024-09-17T07:20:44.369 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:43 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-17T07:20:44.369 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:43 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:44.369 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:43 smithi179 ceph-mon[50085]: Reconfiguring daemon osd.1 on smithi044 2024-09-17T07:20:44.369 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:43 smithi179 ceph-mon[50085]: from='client.44322 172.21.15.44:0/3395778652' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/1355368272"}]: access denied 2024-09-17T07:20:45.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:44 smithi044 ceph-mon[58191]: pgmap v34: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:20:45.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:44 smithi044 ceph-mon[60356]: pgmap v34: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:20:45.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:44 smithi179 ceph-mon[50085]: pgmap v34: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:20:47.257 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:46 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:47.257 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:46 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:47.257 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:46 smithi044 ceph-mon[58191]: Reconfiguring osd.2 (monmap changed)... 2024-09-17T07:20:47.258 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:46 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-17T07:20:47.258 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:46 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:47.258 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:46 smithi044 ceph-mon[58191]: Reconfiguring daemon osd.2 on smithi044 2024-09-17T07:20:47.258 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:46 smithi044 ceph-mon[58191]: pgmap v35: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-09-17T07:20:47.258 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:46 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:47.258 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:46 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:47.258 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:46 smithi044 ceph-mon[60356]: Reconfiguring osd.2 (monmap changed)... 2024-09-17T07:20:47.258 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:46 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-17T07:20:47.258 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:46 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:47.258 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:46 smithi044 ceph-mon[60356]: Reconfiguring daemon osd.2 on smithi044 2024-09-17T07:20:47.258 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:46 smithi044 ceph-mon[60356]: pgmap v35: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-09-17T07:20:47.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:46 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:47.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:46 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:47.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:46 smithi179 ceph-mon[50085]: Reconfiguring osd.2 (monmap changed)... 2024-09-17T07:20:47.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:46 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-17T07:20:47.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:46 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:47.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:46 smithi179 ceph-mon[50085]: Reconfiguring daemon osd.2 on smithi044 2024-09-17T07:20:47.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:46 smithi179 ceph-mon[50085]: pgmap v35: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-09-17T07:20:48.816 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:48 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:48.817 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:48 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:48.817 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:48 smithi044 ceph-mon[58191]: Reconfiguring osd.3 (monmap changed)... 2024-09-17T07:20:48.817 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:48 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-17T07:20:48.817 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:48 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:48.817 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:48 smithi044 ceph-mon[58191]: Reconfiguring daemon osd.3 on smithi044 2024-09-17T07:20:48.817 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:48 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:48.817 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:48 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:48.817 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:48 smithi044 ceph-mon[60356]: Reconfiguring osd.3 (monmap changed)... 2024-09-17T07:20:48.817 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:48 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-17T07:20:48.818 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:48 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:48.818 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:48 smithi044 ceph-mon[60356]: Reconfiguring daemon osd.3 on smithi044 2024-09-17T07:20:48.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:48 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:48.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:48 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:48.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:48 smithi179 ceph-mon[50085]: Reconfiguring osd.3 (monmap changed)... 2024-09-17T07:20:48.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:48 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-17T07:20:48.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:48 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:48.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:48 smithi179 ceph-mon[50085]: Reconfiguring daemon osd.3 on smithi044 2024-09-17T07:20:49.495 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:49 smithi044 ceph-mon[58191]: pgmap v36: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:20:49.495 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:49 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:49.495 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:49 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:49.495 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:49 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi044.nhyhnh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:20:49.495 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:49 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:49.495 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:49 smithi044 ceph-mon[60356]: pgmap v36: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:20:49.495 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:49 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:49.495 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:49 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:49.496 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:49 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi044.nhyhnh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:20:49.496 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:49 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:49.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:49 smithi179 ceph-mon[50085]: pgmap v36: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:20:49.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:49 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:49.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:49 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:49.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:49 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi044.nhyhnh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:20:49.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:49 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:50.574 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:50 smithi044 ceph-mon[60356]: Reconfiguring rgw.foo.smithi044.nhyhnh (monmap changed)... 2024-09-17T07:20:50.574 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:50 smithi044 ceph-mon[60356]: Reconfiguring daemon rgw.foo.smithi044.nhyhnh on smithi044 2024-09-17T07:20:50.574 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:50 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:50.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:50 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:50.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:50 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi044.vnufiz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:20:50.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:50 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:50.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:50 smithi044 ceph-mon[58191]: Reconfiguring rgw.foo.smithi044.nhyhnh (monmap changed)... 2024-09-17T07:20:50.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:50 smithi044 ceph-mon[58191]: Reconfiguring daemon rgw.foo.smithi044.nhyhnh on smithi044 2024-09-17T07:20:50.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:50 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:50.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:50 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:50.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:50 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi044.vnufiz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:20:50.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:50 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:50.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:50 smithi179 ceph-mon[50085]: Reconfiguring rgw.foo.smithi044.nhyhnh (monmap changed)... 2024-09-17T07:20:50.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:50 smithi179 ceph-mon[50085]: Reconfiguring daemon rgw.foo.smithi044.nhyhnh on smithi044 2024-09-17T07:20:50.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:50 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:50.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:50 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:50.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:50 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi044.vnufiz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:20:50.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:50 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:51.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:51 smithi044 ceph-mon[58191]: pgmap v37: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:20:51.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:51 smithi044 ceph-mon[58191]: Reconfiguring rgw.smpl.smithi044.vnufiz (monmap changed)... 2024-09-17T07:20:51.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:51 smithi044 ceph-mon[58191]: Reconfiguring daemon rgw.smpl.smithi044.vnufiz on smithi044 2024-09-17T07:20:51.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:51 smithi044 ceph-mon[60356]: pgmap v37: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:20:51.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:51 smithi044 ceph-mon[60356]: Reconfiguring rgw.smpl.smithi044.vnufiz (monmap changed)... 2024-09-17T07:20:51.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:51 smithi044 ceph-mon[60356]: Reconfiguring daemon rgw.smpl.smithi044.vnufiz on smithi044 2024-09-17T07:20:51.863 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:51 smithi179 ceph-mon[50085]: pgmap v37: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:20:51.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:51 smithi179 ceph-mon[50085]: Reconfiguring rgw.smpl.smithi044.vnufiz (monmap changed)... 2024-09-17T07:20:51.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:51 smithi179 ceph-mon[50085]: Reconfiguring daemon rgw.smpl.smithi044.vnufiz on smithi044 2024-09-17T07:20:53.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:52 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:53.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:52 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:53.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:52 smithi044 ceph-mon[58191]: Reconfiguring mon.b (monmap changed)... 2024-09-17T07:20:53.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:52 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T07:20:53.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:52 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T07:20:53.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:52 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:53.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:52 smithi044 ceph-mon[58191]: Reconfiguring daemon mon.b on smithi179 2024-09-17T07:20:53.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:52 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:53.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:52 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:53.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:52 smithi044 ceph-mon[60356]: Reconfiguring mon.b (monmap changed)... 2024-09-17T07:20:53.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:52 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T07:20:53.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:52 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T07:20:53.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:52 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:53.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:52 smithi044 ceph-mon[60356]: Reconfiguring daemon mon.b on smithi179 2024-09-17T07:20:53.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:52 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:53.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:52 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:53.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:52 smithi179 ceph-mon[50085]: Reconfiguring mon.b (monmap changed)... 2024-09-17T07:20:53.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:52 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T07:20:53.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:52 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T07:20:53.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:52 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:53.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:52 smithi179 ceph-mon[50085]: Reconfiguring daemon mon.b on smithi179 2024-09-17T07:20:53.772 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:53 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Scheduled restart job, restart counter is at 8. 2024-09-17T07:20:53.772 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:53 smithi179 systemd[1]: Stopped Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:20:53.772 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:53 smithi179 systemd[1]: Starting Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:20:54.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:53 smithi044 ceph-mon[58191]: pgmap v38: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:20:54.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:53 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:54.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:53 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:54.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:53 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T07:20:54.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:53 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T07:20:54.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:53 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:54.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:53 smithi044 ceph-mon[60356]: pgmap v38: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:20:54.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:53 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:54.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:53 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:54.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:53 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T07:20:54.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:53 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T07:20:54.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:53 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:54.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:53 smithi179 ceph-mon[50085]: pgmap v38: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:20:54.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:53 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:54.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:53 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:54.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:53 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T07:20:54.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:53 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T07:20:54.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:53 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:54.727 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:54 smithi179 podman[51668]: 2024-09-17 07:20:54.600976908 +0000 UTC m=+0.018344096 image pull de242295e2257c37c8cadfd962369228f8f10b2d48a44259b65fef44ad4f6490 docker.io/prom/prometheus:v2.18.1 2024-09-17T07:20:54.728 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:54 smithi179 podman[51668]: 2024-09-17 07:20:54.727969162 +0000 UTC m=+0.145336340 container create 21318ec4d5a4ec4d00b27721dd552c0d52dd5c826c2a2791ee0003d6c21149f0 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:20:54.988 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:54 smithi179 ceph-mon[50085]: Reconfiguring mgr.x (monmap changed)... 2024-09-17T07:20:54.988 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:54 smithi179 ceph-mon[50085]: Reconfiguring daemon mgr.x on smithi179 2024-09-17T07:20:54.988 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:54 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:54.988 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:54 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:54.988 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:54 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-17T07:20:54.988 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:54 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:54.988 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:54 smithi179 podman[51668]: 2024-09-17 07:20:54.955078075 +0000 UTC m=+0.372445260 container init 21318ec4d5a4ec4d00b27721dd552c0d52dd5c826c2a2791ee0003d6c21149f0 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:20:54.988 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:54 smithi179 podman[51668]: 2024-09-17 07:20:54.957862459 +0000 UTC m=+0.375229646 container start 21318ec4d5a4ec4d00b27721dd552c0d52dd5c826c2a2791ee0003d6c21149f0 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:20:55.036 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:54 smithi044 ceph-mon[58191]: Reconfiguring mgr.x (monmap changed)... 2024-09-17T07:20:55.036 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:54 smithi044 ceph-mon[58191]: Reconfiguring daemon mgr.x on smithi179 2024-09-17T07:20:55.036 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:54 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:55.036 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:54 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:55.037 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:54 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-17T07:20:55.037 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:54 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:55.037 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:54 smithi044 ceph-mon[60356]: Reconfiguring mgr.x (monmap changed)... 2024-09-17T07:20:55.037 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:54 smithi044 ceph-mon[60356]: Reconfiguring daemon mgr.x on smithi179 2024-09-17T07:20:55.037 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:54 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:55.037 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:54 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:55.037 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:54 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-17T07:20:55.037 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:54 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:55.249 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:54.987Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-17T07:20:55.249 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:54.987Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2024-09-17T07:20:55.249 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:54.988Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2024-09-17T07:20:55.249 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:54.988Z caller=main.go:339 host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi179 (none))" 2024-09-17T07:20:55.249 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:54.988Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T07:20:55.250 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:54.988Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T07:20:55.250 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:54.989Z caller=main.go:678 msg="Starting TSDB ..." 2024-09-17T07:20:55.250 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:54.989Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2024-09-17T07:20:55.250 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:54.992Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2024-09-17T07:20:55.250 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:54.992Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=15 2024-09-17T07:20:55.250 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:54.998Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=15 2024-09-17T07:20:55.250 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:54.998Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=15 2024-09-17T07:20:55.250 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:54.998Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=15 2024-09-17T07:20:55.250 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:54 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:54.999Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=15 2024-09-17T07:20:55.250 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:55.001Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=15 2024-09-17T07:20:55.250 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:55.001Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=15 2024-09-17T07:20:55.250 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:55.001Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=7 maxSegment=15 2024-09-17T07:20:55.250 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:55.001Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=8 maxSegment=15 2024-09-17T07:20:55.251 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:55.001Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=9 maxSegment=15 2024-09-17T07:20:55.251 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:55.001Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=10 maxSegment=15 2024-09-17T07:20:55.251 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:55.001Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=11 maxSegment=15 2024-09-17T07:20:55.251 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:55.001Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=12 maxSegment=15 2024-09-17T07:20:55.251 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:55.001Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=13 maxSegment=15 2024-09-17T07:20:55.251 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:55.002Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=14 maxSegment=15 2024-09-17T07:20:55.251 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:55.002Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=15 maxSegment=15 2024-09-17T07:20:55.252 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:55.002Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=9.438422ms 2024-09-17T07:20:55.252 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:55.003Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2024-09-17T07:20:55.252 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:55.003Z caller=main.go:695 msg="TSDB started" 2024-09-17T07:20:55.252 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:55.003Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T07:20:55.252 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:55.004Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2024-09-17T07:20:55.252 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:55.004Z caller=main.go:561 msg="Stopping notify discovery manager..." 2024-09-17T07:20:55.252 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:55.004Z caller=main.go:583 msg="Stopping scrape manager..." 2024-09-17T07:20:55.252 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:55.004Z caller=main.go:557 msg="Notify discovery manager stopped" 2024-09-17T07:20:55.252 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:55.004Z caller=main.go:543 msg="Scrape discovery manager stopped" 2024-09-17T07:20:55.252 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:55.004Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2024-09-17T07:20:55.252 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:55.004Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2024-09-17T07:20:55.252 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:55.004Z caller=main.go:577 msg="Scrape manager stopped" 2024-09-17T07:20:55.252 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:55.004Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2024-09-17T07:20:55.253 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=info ts=2024-09-17T07:20:55.004Z caller=main.go:749 msg="Notifier manager stopped" 2024-09-17T07:20:55.253 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:55 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[51743]: level=error ts=2024-09-17T07:20:55.004Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 14: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain\n line 31: field http_sd_configs not found in type config.plain\n line 45: field http_sd_configs not found in type config.plain\n line 49: field http_sd_configs not found in type config.plain" 2024-09-17T07:20:55.253 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:55 smithi179 bash[51668]: 21318ec4d5a4ec4d00b27721dd552c0d52dd5c826c2a2791ee0003d6c21149f0 2024-09-17T07:20:55.253 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:55 smithi179 systemd[1]: Started Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:20:55.253 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:55 smithi179 podman[51776]: 2024-09-17 07:20:55.062900916 +0000 UTC m=+0.050180898 container died 21318ec4d5a4ec4d00b27721dd552c0d52dd5c826c2a2791ee0003d6c21149f0 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:20:55.541 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-09-17T07:20:55.778 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:55 smithi179 podman[51776]: 2024-09-17 07:20:55.677038624 +0000 UTC m=+0.664318598 container remove 21318ec4d5a4ec4d00b27721dd552c0d52dd5c826c2a2791ee0003d6c21149f0 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:20:55.778 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:55 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2024-09-17T07:20:55.778 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:20:55 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Failed with result 'exit-code'. 2024-09-17T07:20:55.973 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:55 smithi044 ceph-mon[60356]: pgmap v39: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:20:55.973 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:55 smithi044 ceph-mon[60356]: Reconfiguring osd.4 (monmap changed)... 2024-09-17T07:20:55.973 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:55 smithi044 ceph-mon[60356]: Reconfiguring daemon osd.4 on smithi179 2024-09-17T07:20:55.973 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:55 smithi044 ceph-mon[58191]: pgmap v39: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:20:55.973 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:55 smithi044 ceph-mon[58191]: Reconfiguring osd.4 (monmap changed)... 2024-09-17T07:20:55.973 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:55 smithi044 ceph-mon[58191]: Reconfiguring daemon osd.4 on smithi179 2024-09-17T07:20:56.098 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:55 smithi179 ceph-mon[50085]: pgmap v39: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:20:56.098 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:55 smithi179 ceph-mon[50085]: Reconfiguring osd.4 (monmap changed)... 2024-09-17T07:20:56.098 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:55 smithi179 ceph-mon[50085]: Reconfiguring daemon osd.4 on smithi179 2024-09-17T07:20:56.205 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T07:20:56.206 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.a smithi044 *:9093,9094 running (2m) 41s ago 7m 16.8M - 0.20.0 0881eb8f169f 0473d4b023ad 2024-09-17T07:20:56.206 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.a smithi179 *:3000 running (2m) 23s ago 7m 27.2M - 6.7.4 557c83e11646 43ab0e03dcac 2024-09-17T07:20:56.206 INFO:teuthology.orchestra.run.smithi044.stdout:iscsi.foo.smithi044.elmtfk smithi044 unknown 41s ago 6m - - 24ecd6d5f14c 17a7859c6333 2024-09-17T07:20:56.206 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.x smithi179 *:8443,8765 running (87s) 23s ago 11m 436M - 18.2.4-855-g19e2a2be 980a0bfcd684 66b52dc26022 2024-09-17T07:20:56.206 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.y smithi044 *:8443,8765 running (2m) 41s ago 12m 471M - 18.2.4-855-g19e2a2be 980a0bfcd684 8080dd318a1c 2024-09-17T07:20:56.206 INFO:teuthology.orchestra.run.smithi044.stdout:mon.a smithi044 running (73s) 41s ago 12m 38.6M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 c3293e8cdbba 2024-09-17T07:20:56.206 INFO:teuthology.orchestra.run.smithi044.stdout:mon.b smithi179 running (31s) 23s ago 11m 27.7M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 ade749759184 2024-09-17T07:20:56.206 INFO:teuthology.orchestra.run.smithi044.stdout:mon.c smithi044 running (49s) 41s ago 11m 23.5M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 02f85d31e0bd 2024-09-17T07:20:56.206 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.a smithi044 *:9100 running (2m) 41s ago 7m 17.0M - 0.18.1 e5a616e4b9cf 8a2cde5bd9ed 2024-09-17T07:20:56.206 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.b smithi179 *:9100 running (2m) 23s ago 8m 8929k - 0.18.1 e5a616e4b9cf 04c63b5cb524 2024-09-17T07:20:56.206 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (11m) 41s ago 11m 48.4M 4096M 16.2.0 24ecd6d5f14c 4517de6029ac 2024-09-17T07:20:56.206 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (10m) 41s ago 10m 45.8M 4096M 16.2.0 24ecd6d5f14c 2d5122cbeff9 2024-09-17T07:20:56.207 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (10m) 41s ago 10m 43.4M 4096M 16.2.0 24ecd6d5f14c 7fd2f30be874 2024-09-17T07:20:56.207 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi044 running (10m) 41s ago 10m 47.7M 4096M 16.2.0 24ecd6d5f14c 2b7d623395a7 2024-09-17T07:20:56.207 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi179 running (9m) 23s ago 9m 51.7M 4096M 16.2.0 24ecd6d5f14c 911e6f47c822 2024-09-17T07:20:56.207 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi179 running (9m) 23s ago 9m 47.8M 4096M 16.2.0 24ecd6d5f14c e56060742dff 2024-09-17T07:20:56.207 INFO:teuthology.orchestra.run.smithi044.stdout:osd.6 smithi179 running (9m) 23s ago 9m 47.4M 4096M 16.2.0 24ecd6d5f14c 7a5c30b68cbc 2024-09-17T07:20:56.207 INFO:teuthology.orchestra.run.smithi044.stdout:osd.7 smithi179 running (8m) 23s ago 8m 53.5M 4096M 16.2.0 24ecd6d5f14c b0d186fbf8a6 2024-09-17T07:20:56.207 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.a smithi179 *:9095 unknown 23s ago 8m - - 2024-09-17T07:20:56.207 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi044.nhyhnh smithi044 *:8000 running (6m) 41s ago 6m 38.5M - 16.2.0 24ecd6d5f14c 9b455479614f 2024-09-17T07:20:56.207 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi179.tqlzyy smithi179 *:8000 running (6m) 23s ago 6m 41.2M - 16.2.0 24ecd6d5f14c c8d1693241ce 2024-09-17T07:20:56.207 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi044.vnufiz smithi044 *:80 running (6m) 41s ago 6m 38.2M - 16.2.0 24ecd6d5f14c 6eae2bff5ec8 2024-09-17T07:20:56.207 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi179.ecjhll smithi179 *:80 running (6m) 23s ago 6m 41.7M - 16.2.0 24ecd6d5f14c 253109ee0394 2024-09-17T07:20:56.620 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:20:56.620 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-09-17T07:20:56.620 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3 2024-09-17T07:20:56.620 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:20:56.620 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-09-17T07:20:56.620 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T07:20:56.621 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:20:56.621 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-09-17T07:20:56.621 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2024-09-17T07:20:56.621 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:20:56.621 INFO:teuthology.orchestra.run.smithi044.stdout: "rgw": { 2024-09-17T07:20:56.621 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2024-09-17T07:20:56.621 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:20:56.621 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-09-17T07:20:56.621 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 12, 2024-09-17T07:20:56.621 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 5 2024-09-17T07:20:56.621 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-09-17T07:20:56.621 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:20:56.978 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:56 smithi044 ceph-mon[58191]: from='client.54211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:20:56.978 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:56 smithi044 ceph-mon[58191]: from='client.44334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:20:56.978 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:56 smithi044 ceph-mon[58191]: from='client.54247 172.21.15.44:0/1268761450' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:20:56.978 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:56 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/25358579' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:20:56.978 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:56 smithi044 ceph-mon[60356]: from='client.54211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:20:56.978 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:56 smithi044 ceph-mon[60356]: from='client.44334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:20:56.978 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:56 smithi044 ceph-mon[60356]: from='client.54247 172.21.15.44:0/1268761450' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:20:56.978 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:56 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/25358579' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:20:57.013 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:20:57.013 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9", 2024-09-17T07:20:57.013 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-09-17T07:20:57.013 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T07:20:57.013 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [ 2024-09-17T07:20:57.013 INFO:teuthology.orchestra.run.smithi044.stdout: "mon", 2024-09-17T07:20:57.013 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr" 2024-09-17T07:20:57.014 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-09-17T07:20:57.014 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "5/23 daemons upgraded", 2024-09-17T07:20:57.014 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "Currently upgrading mon daemons", 2024-09-17T07:20:57.014 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-09-17T07:20:57.014 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:20:57.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:56 smithi179 ceph-mon[50085]: from='client.54211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:20:57.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:56 smithi179 ceph-mon[50085]: from='client.44334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:20:57.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:56 smithi179 ceph-mon[50085]: from='client.54247 172.21.15.44:0/1268761450' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:20:57.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:56 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/25358579' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:20:57.421 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_WARN 2 failed cephadm daemon(s) 2024-09-17T07:20:57.421 INFO:teuthology.orchestra.run.smithi044.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T07:20:57.421 INFO:teuthology.orchestra.run.smithi044.stdout: daemon iscsi.foo.smithi044.elmtfk on smithi044 is in unknown state 2024-09-17T07:20:57.422 INFO:teuthology.orchestra.run.smithi044.stdout: daemon prometheus.a on smithi179 is in unknown state 2024-09-17T07:20:58.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:57 smithi044 ceph-mon[58191]: from='client.64139 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:20:58.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:57 smithi044 ceph-mon[58191]: pgmap v40: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:20:58.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:57 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:58.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:57 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:58.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:57 smithi044 ceph-mon[58191]: Reconfiguring osd.5 (monmap changed)... 2024-09-17T07:20:58.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:57 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-17T07:20:58.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:57 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:58.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:57 smithi044 ceph-mon[58191]: Reconfiguring daemon osd.5 on smithi179 2024-09-17T07:20:58.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:57 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/310673744' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:20:58.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:57 smithi044 ceph-mon[58191]: from='client.44382 172.21.15.44:0/923705202' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:20:58.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:57 smithi044 ceph-mon[60356]: from='client.64139 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:20:58.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:57 smithi044 ceph-mon[60356]: pgmap v40: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:20:58.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:57 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:58.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:57 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:58.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:57 smithi044 ceph-mon[60356]: Reconfiguring osd.5 (monmap changed)... 2024-09-17T07:20:58.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:57 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-17T07:20:58.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:57 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:58.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:57 smithi044 ceph-mon[60356]: Reconfiguring daemon osd.5 on smithi179 2024-09-17T07:20:58.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:57 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/310673744' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:20:58.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:57 smithi044 ceph-mon[60356]: from='client.44382 172.21.15.44:0/923705202' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:20:58.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:57 smithi179 ceph-mon[50085]: from='client.64139 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:20:58.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:57 smithi179 ceph-mon[50085]: pgmap v40: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:20:58.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:57 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:58.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:57 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:58.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:57 smithi179 ceph-mon[50085]: Reconfiguring osd.5 (monmap changed)... 2024-09-17T07:20:58.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:57 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-17T07:20:58.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:57 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:58.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:57 smithi179 ceph-mon[50085]: Reconfiguring daemon osd.5 on smithi179 2024-09-17T07:20:58.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:57 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/310673744' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:20:58.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:57 smithi179 ceph-mon[50085]: from='client.44382 172.21.15.44:0/923705202' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:20:59.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:58 smithi179 ceph-mon[50085]: from='client.44376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:20:59.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:58 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:59.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:58 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:59.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:58 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-17T07:20:59.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:58 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:59.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:58 smithi044 ceph-mon[58191]: from='client.44376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:20:59.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:58 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:59.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:58 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:59.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:58 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-17T07:20:59.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:58 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:20:59.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:58 smithi044 ceph-mon[60356]: from='client.44376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:20:59.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:58 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:59.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:58 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:20:59.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:58 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-17T07:20:59.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:58 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:21:00.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:59 smithi179 ceph-mon[50085]: Reconfiguring osd.6 (monmap changed)... 2024-09-17T07:21:00.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:59 smithi179 ceph-mon[50085]: Reconfiguring daemon osd.6 on smithi179 2024-09-17T07:21:00.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:59 smithi179 ceph-mon[50085]: pgmap v41: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:21:00.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:59 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:00.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:59 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:00.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:59 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-17T07:21:00.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:20:59 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:21:00.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:59 smithi044 ceph-mon[58191]: Reconfiguring osd.6 (monmap changed)... 2024-09-17T07:21:00.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:59 smithi044 ceph-mon[58191]: Reconfiguring daemon osd.6 on smithi179 2024-09-17T07:21:00.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:59 smithi044 ceph-mon[58191]: pgmap v41: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:21:00.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:59 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:00.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:59 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:00.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:59 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-17T07:21:00.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:20:59 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:21:00.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:59 smithi044 ceph-mon[60356]: Reconfiguring osd.6 (monmap changed)... 2024-09-17T07:21:00.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:59 smithi044 ceph-mon[60356]: Reconfiguring daemon osd.6 on smithi179 2024-09-17T07:21:00.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:59 smithi044 ceph-mon[60356]: pgmap v41: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:21:00.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:59 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:00.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:59 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:00.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:59 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-17T07:21:00.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:20:59 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:21:01.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:00 smithi179 ceph-mon[50085]: Reconfiguring osd.7 (monmap changed)... 2024-09-17T07:21:01.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:00 smithi179 ceph-mon[50085]: Reconfiguring daemon osd.7 on smithi179 2024-09-17T07:21:01.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:00 smithi044 ceph-mon[58191]: Reconfiguring osd.7 (monmap changed)... 2024-09-17T07:21:01.327 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:00 smithi044 ceph-mon[58191]: Reconfiguring daemon osd.7 on smithi179 2024-09-17T07:21:01.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:00 smithi044 ceph-mon[60356]: Reconfiguring osd.7 (monmap changed)... 2024-09-17T07:21:01.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:00 smithi044 ceph-mon[60356]: Reconfiguring daemon osd.7 on smithi179 2024-09-17T07:21:02.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:01 smithi179 ceph-mon[50085]: pgmap v42: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:21:02.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:01 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:02.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:01 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:02.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:01 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi179.tqlzyy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:21:02.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:01 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:21:02.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:01 smithi044 ceph-mon[58191]: pgmap v42: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:21:02.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:01 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:02.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:01 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:02.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:01 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi179.tqlzyy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:21:02.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:01 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:21:02.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:01 smithi044 ceph-mon[60356]: pgmap v42: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:21:02.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:01 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:02.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:01 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:02.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:01 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi179.tqlzyy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:21:02.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:01 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:21:03.108 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:02 smithi179 ceph-mon[50085]: Reconfiguring rgw.foo.smithi179.tqlzyy (monmap changed)... 2024-09-17T07:21:03.108 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:02 smithi179 ceph-mon[50085]: Reconfiguring daemon rgw.foo.smithi179.tqlzyy on smithi179 2024-09-17T07:21:03.109 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:02 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:03.109 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:02 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:03.109 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:02 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi179.ecjhll", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:21:03.109 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:02 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:21:03.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:02 smithi044 ceph-mon[58191]: Reconfiguring rgw.foo.smithi179.tqlzyy (monmap changed)... 2024-09-17T07:21:03.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:02 smithi044 ceph-mon[58191]: Reconfiguring daemon rgw.foo.smithi179.tqlzyy on smithi179 2024-09-17T07:21:03.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:02 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:03.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:02 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:03.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:02 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi179.ecjhll", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:21:03.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:02 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:21:03.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:02 smithi044 ceph-mon[60356]: Reconfiguring rgw.foo.smithi179.tqlzyy (monmap changed)... 2024-09-17T07:21:03.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:02 smithi044 ceph-mon[60356]: Reconfiguring daemon rgw.foo.smithi179.tqlzyy on smithi179 2024-09-17T07:21:03.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:02 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:03.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:02 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:03.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:02 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi179.ecjhll", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:21:03.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:02 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:21:04.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:03 smithi179 ceph-mon[50085]: pgmap v43: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:21:04.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:03 smithi179 ceph-mon[50085]: Reconfiguring rgw.smpl.smithi179.ecjhll (monmap changed)... 2024-09-17T07:21:04.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:03 smithi179 ceph-mon[50085]: Reconfiguring daemon rgw.smpl.smithi179.ecjhll on smithi179 2024-09-17T07:21:04.299 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:03 smithi044 ceph-mon[58191]: pgmap v43: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:21:04.299 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:03 smithi044 ceph-mon[58191]: Reconfiguring rgw.smpl.smithi179.ecjhll (monmap changed)... 2024-09-17T07:21:04.299 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:03 smithi044 ceph-mon[58191]: Reconfiguring daemon rgw.smpl.smithi179.ecjhll on smithi179 2024-09-17T07:21:04.299 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:03 smithi044 ceph-mon[60356]: pgmap v43: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:21:04.299 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:03 smithi044 ceph-mon[60356]: Reconfiguring rgw.smpl.smithi179.ecjhll (monmap changed)... 2024-09-17T07:21:04.300 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:03 smithi044 ceph-mon[60356]: Reconfiguring daemon rgw.smpl.smithi179.ecjhll on smithi179 2024-09-17T07:21:05.249 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:04 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:05.249 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:04 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:05.249 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:04 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:21:05.249 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:04 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:21:05.249 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:04 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:21:05.249 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:04 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:05.249 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:04 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-17T07:21:05.249 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:04 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-17T07:21:05.249 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:04 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-09-17T07:21:05.249 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:04 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-17T07:21:05.250 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:04 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-17T07:21:05.250 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:04 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-17T07:21:05.250 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:04 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:21:05.250 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:04 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:05.250 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:04 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-17T07:21:05.250 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:04 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:05.250 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:04 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:05.250 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:04 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:21:05.250 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:04 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:21:05.250 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:04 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:21:05.250 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:04 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:05.251 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:04 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-17T07:21:05.251 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:04 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-17T07:21:05.251 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:04 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-09-17T07:21:05.251 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:04 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-17T07:21:05.251 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:04 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-17T07:21:05.251 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:04 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-17T07:21:05.251 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:04 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:21:05.251 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:04 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:05.251 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:04 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-17T07:21:05.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:04 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:05.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:04 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:05.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:04 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:21:05.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:04 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:21:05.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:04 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:21:05.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:04 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:05.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:04 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-17T07:21:05.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:04 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-17T07:21:05.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:04 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-09-17T07:21:05.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:04 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-17T07:21:05.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:04 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-17T07:21:05.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:04 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-17T07:21:05.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:04 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:21:05.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:04 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:05.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:04 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-17T07:21:06.027 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:05 smithi179 ceph-mon[50085]: Upgrade: Setting container_image for all mon 2024-09-17T07:21:06.027 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:05 smithi179 ceph-mon[50085]: Upgrade: Setting container_image for all crash 2024-09-17T07:21:06.027 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:05 smithi179 ceph-mon[50085]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-17T07:21:06.028 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:05 smithi179 ceph-mon[50085]: Upgrade: osd.0 is safe to restart 2024-09-17T07:21:06.028 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:05 smithi179 ceph-mon[50085]: pgmap v44: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:21:06.028 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:05 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:06.028 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:05 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-17T07:21:06.028 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:05 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:21:06.028 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:05 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Scheduled restart job, restart counter is at 9. 2024-09-17T07:21:06.028 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:05 smithi179 systemd[1]: Stopped Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:21:06.028 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:05 smithi179 systemd[1]: Starting Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:21:06.028 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:05 smithi179 podman[52513]: 2024-09-17 07:21:05.803577888 +0000 UTC m=+0.017438008 image pull de242295e2257c37c8cadfd962369228f8f10b2d48a44259b65fef44ad4f6490 docker.io/prom/prometheus:v2.18.1 2024-09-17T07:21:06.028 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:05 smithi179 podman[52513]: 2024-09-17 07:21:05.93791195 +0000 UTC m=+0.151772068 container create d27b9b884886705114cddf719d4d74109f7890aa703acc958833d0f54d8fef3e (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:21:06.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:05 smithi044 ceph-mon[60356]: Upgrade: Setting container_image for all mon 2024-09-17T07:21:06.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:05 smithi044 ceph-mon[60356]: Upgrade: Setting container_image for all crash 2024-09-17T07:21:06.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:05 smithi044 ceph-mon[60356]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-17T07:21:06.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:05 smithi044 ceph-mon[60356]: Upgrade: osd.0 is safe to restart 2024-09-17T07:21:06.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:05 smithi044 ceph-mon[60356]: pgmap v44: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:21:06.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:05 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:06.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:05 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-17T07:21:06.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:05 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:21:06.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:05 smithi044 ceph-mon[58191]: Upgrade: Setting container_image for all mon 2024-09-17T07:21:06.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:05 smithi044 ceph-mon[58191]: Upgrade: Setting container_image for all crash 2024-09-17T07:21:06.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:05 smithi044 ceph-mon[58191]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-17T07:21:06.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:05 smithi044 ceph-mon[58191]: Upgrade: osd.0 is safe to restart 2024-09-17T07:21:06.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:05 smithi044 ceph-mon[58191]: pgmap v44: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:21:06.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:05 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:06.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:05 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-17T07:21:06.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:05 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:21:06.364 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 podman[52513]: 2024-09-17 07:21:06.131266449 +0000 UTC m=+0.345126570 container init d27b9b884886705114cddf719d4d74109f7890aa703acc958833d0f54d8fef3e (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:21:06.364 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 podman[52513]: 2024-09-17 07:21:06.134604034 +0000 UTC m=+0.348464161 container start d27b9b884886705114cddf719d4d74109f7890aa703acc958833d0f54d8fef3e (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:21:06.364 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.164Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-17T07:21:06.364 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.164Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2024-09-17T07:21:06.364 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.164Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2024-09-17T07:21:06.364 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.164Z caller=main.go:339 host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi179 (none))" 2024-09-17T07:21:06.364 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.164Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T07:21:06.364 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.164Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T07:21:06.365 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.166Z caller=main.go:678 msg="Starting TSDB ..." 2024-09-17T07:21:06.365 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.166Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2024-09-17T07:21:06.365 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.168Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2024-09-17T07:21:06.365 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.169Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=16 2024-09-17T07:21:06.365 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.176Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=16 2024-09-17T07:21:06.365 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.177Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=16 2024-09-17T07:21:06.365 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.177Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=16 2024-09-17T07:21:06.365 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.177Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=16 2024-09-17T07:21:06.365 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.177Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=16 2024-09-17T07:21:06.366 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.177Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=16 2024-09-17T07:21:06.366 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.177Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=7 maxSegment=16 2024-09-17T07:21:06.366 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.177Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=8 maxSegment=16 2024-09-17T07:21:06.366 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.178Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=9 maxSegment=16 2024-09-17T07:21:06.366 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.178Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=10 maxSegment=16 2024-09-17T07:21:06.366 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.178Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=11 maxSegment=16 2024-09-17T07:21:06.366 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.178Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=12 maxSegment=16 2024-09-17T07:21:06.366 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.178Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=13 maxSegment=16 2024-09-17T07:21:06.366 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.179Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=14 maxSegment=16 2024-09-17T07:21:06.366 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.180Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=15 maxSegment=16 2024-09-17T07:21:06.366 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.180Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=16 maxSegment=16 2024-09-17T07:21:06.366 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.180Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=12.070364ms 2024-09-17T07:21:06.367 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.182Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2024-09-17T07:21:06.367 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.182Z caller=main.go:695 msg="TSDB started" 2024-09-17T07:21:06.367 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.182Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T07:21:06.367 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.183Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2024-09-17T07:21:06.367 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.183Z caller=main.go:561 msg="Stopping notify discovery manager..." 2024-09-17T07:21:06.367 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.183Z caller=main.go:583 msg="Stopping scrape manager..." 2024-09-17T07:21:06.367 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.183Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2024-09-17T07:21:06.367 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.183Z caller=main.go:557 msg="Notify discovery manager stopped" 2024-09-17T07:21:06.367 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.183Z caller=main.go:543 msg="Scrape discovery manager stopped" 2024-09-17T07:21:06.367 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.183Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2024-09-17T07:21:06.367 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.183Z caller=main.go:577 msg="Scrape manager stopped" 2024-09-17T07:21:06.367 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.183Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2024-09-17T07:21:06.367 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=info ts=2024-09-17T07:21:06.183Z caller=main.go:749 msg="Notifier manager stopped" 2024-09-17T07:21:06.367 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52530]: level=error ts=2024-09-17T07:21:06.183Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 14: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain\n line 31: field http_sd_configs not found in type config.plain\n line 45: field http_sd_configs not found in type config.plain\n line 49: field http_sd_configs not found in type config.plain" 2024-09-17T07:21:06.368 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 bash[52513]: d27b9b884886705114cddf719d4d74109f7890aa703acc958833d0f54d8fef3e 2024-09-17T07:21:06.368 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 systemd[1]: Started Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:21:06.368 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 podman[52547]: 2024-09-17 07:21:06.231848722 +0000 UTC m=+0.037191620 container died d27b9b884886705114cddf719d4d74109f7890aa703acc958833d0f54d8fef3e (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:21:07.110 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:06 smithi044 ceph-mon[58191]: Upgrade: Updating osd.0 2024-09-17T07:21:07.110 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:06 smithi044 ceph-mon[58191]: Deploying daemon osd.0 on smithi044 2024-09-17T07:21:07.110 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:06 smithi044 ceph-mon[60356]: Upgrade: Updating osd.0 2024-09-17T07:21:07.111 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:06 smithi044 ceph-mon[60356]: Deploying daemon osd.0 on smithi044 2024-09-17T07:21:07.113 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-mon[50085]: Upgrade: Updating osd.0 2024-09-17T07:21:07.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:06 smithi179 ceph-mon[50085]: Deploying daemon osd.0 on smithi044 2024-09-17T07:21:07.114 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 podman[52547]: 2024-09-17 07:21:06.809999898 +0000 UTC m=+0.615342788 container remove d27b9b884886705114cddf719d4d74109f7890aa703acc958833d0f54d8fef3e (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:21:07.114 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2024-09-17T07:21:07.114 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:06 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Failed with result 'exit-code'. 2024-09-17T07:21:08.156 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:07 smithi044 ceph-mon[58191]: pgmap v45: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:21:08.156 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:07 smithi044 ceph-mon[60356]: pgmap v45: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:21:08.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:07 smithi179 ceph-mon[50085]: pgmap v45: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:21:08.521 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:08 smithi044 systemd[1]: Stopping Ceph osd.0 for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:21:08.824 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.0[29263]: debug 2024-09-17T07:21:08.519+0000 7f6d94007700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-17T07:21:08.824 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.0[29263]: debug 2024-09-17T07:21:08.519+0000 7f6d94007700 -1 osd.0 76 *** Got signal Terminated *** 2024-09-17T07:21:08.824 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.0[29263]: debug 2024-09-17T07:21:08.519+0000 7f6d94007700 -1 osd.0 76 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T07:21:09.180 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:09 smithi044 podman[64296]: 2024-09-17 07:21:09.179666774 +0000 UTC m=+0.778933527 container died 4517de6029ac38c2976470420d78c9ee0372a5a63e66bf0fbcf0fe6b8dfb70bb (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.0, org.label-schema.build-date=20201204, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-16.2.0, maintainer=Dimitri Savineau , RELEASE=HEAD, org.label-schema.name=CentOS Base Image, GIT_CLEAN=True, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, ceph=True) 2024-09-17T07:21:09.181 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[58191]: osd.0 reported immediately failed by osd.5 2024-09-17T07:21:09.181 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[58191]: osd.0 failed (root=default,host=smithi044) (connection refused reported by osd.5) 2024-09-17T07:21:09.181 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[58191]: osd.0 reported immediately failed by osd.5 2024-09-17T07:21:09.181 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[58191]: osd.0 reported immediately failed by osd.1 2024-09-17T07:21:09.181 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[58191]: osd.0 reported immediately failed by osd.1 2024-09-17T07:21:09.181 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[58191]: osd.0 reported immediately failed by osd.3 2024-09-17T07:21:09.181 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[58191]: osd.0 reported immediately failed by osd.3 2024-09-17T07:21:09.181 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[58191]: osd.0 reported immediately failed by osd.6 2024-09-17T07:21:09.181 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[58191]: osd.0 reported immediately failed by osd.5 2024-09-17T07:21:09.181 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[58191]: osd.0 reported immediately failed by osd.4 2024-09-17T07:21:09.181 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[58191]: osd.0 reported immediately failed by osd.2 2024-09-17T07:21:09.182 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[58191]: osd.0 reported immediately failed by osd.6 2024-09-17T07:21:09.182 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[58191]: osd.0 reported immediately failed by osd.5 2024-09-17T07:21:09.182 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[58191]: osd.0 reported immediately failed by osd.4 2024-09-17T07:21:09.182 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[58191]: osd.0 reported immediately failed by osd.2 2024-09-17T07:21:09.182 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[58191]: osd.0 reported immediately failed by osd.7 2024-09-17T07:21:09.182 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[58191]: osd.0 reported immediately failed by osd.7 2024-09-17T07:21:09.182 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[58191]: osd.0 reported immediately failed by osd.4 2024-09-17T07:21:09.182 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[58191]: osd.0 reported immediately failed by osd.6 2024-09-17T07:21:09.182 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[58191]: osd.0 reported immediately failed by osd.3 2024-09-17T07:21:09.182 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[58191]: osd.0 reported immediately failed by osd.7 2024-09-17T07:21:09.183 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[60356]: osd.0 reported immediately failed by osd.5 2024-09-17T07:21:09.183 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[60356]: osd.0 failed (root=default,host=smithi044) (connection refused reported by osd.5) 2024-09-17T07:21:09.183 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[60356]: osd.0 reported immediately failed by osd.5 2024-09-17T07:21:09.183 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[60356]: osd.0 reported immediately failed by osd.1 2024-09-17T07:21:09.183 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[60356]: osd.0 reported immediately failed by osd.1 2024-09-17T07:21:09.183 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[60356]: osd.0 reported immediately failed by osd.3 2024-09-17T07:21:09.183 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[60356]: osd.0 reported immediately failed by osd.3 2024-09-17T07:21:09.183 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[60356]: osd.0 reported immediately failed by osd.6 2024-09-17T07:21:09.183 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[60356]: osd.0 reported immediately failed by osd.5 2024-09-17T07:21:09.183 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[60356]: osd.0 reported immediately failed by osd.4 2024-09-17T07:21:09.183 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[60356]: osd.0 reported immediately failed by osd.2 2024-09-17T07:21:09.183 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[60356]: osd.0 reported immediately failed by osd.6 2024-09-17T07:21:09.184 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[60356]: osd.0 reported immediately failed by osd.5 2024-09-17T07:21:09.184 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[60356]: osd.0 reported immediately failed by osd.4 2024-09-17T07:21:09.184 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[60356]: osd.0 reported immediately failed by osd.2 2024-09-17T07:21:09.184 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[60356]: osd.0 reported immediately failed by osd.7 2024-09-17T07:21:09.184 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[60356]: osd.0 reported immediately failed by osd.7 2024-09-17T07:21:09.184 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[60356]: osd.0 reported immediately failed by osd.4 2024-09-17T07:21:09.184 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[60356]: osd.0 reported immediately failed by osd.6 2024-09-17T07:21:09.184 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[60356]: osd.0 reported immediately failed by osd.3 2024-09-17T07:21:09.184 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:08 smithi044 ceph-mon[60356]: osd.0 reported immediately failed by osd.7 2024-09-17T07:21:09.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:08 smithi179 ceph-mon[50085]: osd.0 reported immediately failed by osd.5 2024-09-17T07:21:09.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:08 smithi179 ceph-mon[50085]: osd.0 failed (root=default,host=smithi044) (connection refused reported by osd.5) 2024-09-17T07:21:09.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:08 smithi179 ceph-mon[50085]: osd.0 reported immediately failed by osd.5 2024-09-17T07:21:09.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:08 smithi179 ceph-mon[50085]: osd.0 reported immediately failed by osd.1 2024-09-17T07:21:09.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:08 smithi179 ceph-mon[50085]: osd.0 reported immediately failed by osd.1 2024-09-17T07:21:09.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:08 smithi179 ceph-mon[50085]: osd.0 reported immediately failed by osd.3 2024-09-17T07:21:09.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:08 smithi179 ceph-mon[50085]: osd.0 reported immediately failed by osd.3 2024-09-17T07:21:09.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:08 smithi179 ceph-mon[50085]: osd.0 reported immediately failed by osd.6 2024-09-17T07:21:09.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:08 smithi179 ceph-mon[50085]: osd.0 reported immediately failed by osd.5 2024-09-17T07:21:09.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:08 smithi179 ceph-mon[50085]: osd.0 reported immediately failed by osd.4 2024-09-17T07:21:09.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:08 smithi179 ceph-mon[50085]: osd.0 reported immediately failed by osd.2 2024-09-17T07:21:09.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:08 smithi179 ceph-mon[50085]: osd.0 reported immediately failed by osd.6 2024-09-17T07:21:09.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:08 smithi179 ceph-mon[50085]: osd.0 reported immediately failed by osd.5 2024-09-17T07:21:09.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:08 smithi179 ceph-mon[50085]: osd.0 reported immediately failed by osd.4 2024-09-17T07:21:09.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:08 smithi179 ceph-mon[50085]: osd.0 reported immediately failed by osd.2 2024-09-17T07:21:09.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:08 smithi179 ceph-mon[50085]: osd.0 reported immediately failed by osd.7 2024-09-17T07:21:09.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:08 smithi179 ceph-mon[50085]: osd.0 reported immediately failed by osd.7 2024-09-17T07:21:09.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:08 smithi179 ceph-mon[50085]: osd.0 reported immediately failed by osd.4 2024-09-17T07:21:09.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:08 smithi179 ceph-mon[50085]: osd.0 reported immediately failed by osd.6 2024-09-17T07:21:09.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:08 smithi179 ceph-mon[50085]: osd.0 reported immediately failed by osd.3 2024-09-17T07:21:09.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:08 smithi179 ceph-mon[50085]: osd.0 reported immediately failed by osd.7 2024-09-17T07:21:09.774 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:09 smithi044 podman[64296]: 2024-09-17 07:21:09.581581955 +0000 UTC m=+1.180848707 container cleanup 4517de6029ac38c2976470420d78c9ee0372a5a63e66bf0fbcf0fe6b8dfb70bb (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.0, CEPH_POINT_RELEASE=-16.2.0, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, ceph=True, maintainer=Dimitri Savineau , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20201204, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image) 2024-09-17T07:21:09.774 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:09 smithi044 bash[64296]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.0 2024-09-17T07:21:10.060 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:09 smithi044 podman[64348]: 2024-09-17 07:21:09.774642245 +0000 UTC m=+1.221142505 container remove 4517de6029ac38c2976470420d78c9ee0372a5a63e66bf0fbcf0fe6b8dfb70bb (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.0, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, RELEASE=HEAD, maintainer=Dimitri Savineau , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-16.2.0, org.label-schema.schema-version=1.0) 2024-09-17T07:21:10.060 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:09 smithi044 ceph-mon[58191]: pgmap v46: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:21:10.060 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:09 smithi044 ceph-mon[58191]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T07:21:10.060 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:09 smithi044 ceph-mon[58191]: osdmap e77: 8 total, 7 up, 8 in 2024-09-17T07:21:10.060 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:09 smithi044 ceph-mon[60356]: pgmap v46: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:21:10.060 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:09 smithi044 ceph-mon[60356]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T07:21:10.060 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:09 smithi044 ceph-mon[60356]: osdmap e77: 8 total, 7 up, 8 in 2024-09-17T07:21:10.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:09 smithi179 ceph-mon[50085]: pgmap v46: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:21:10.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:09 smithi179 ceph-mon[50085]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T07:21:10.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:09 smithi179 ceph-mon[50085]: osdmap e77: 8 total, 7 up, 8 in 2024-09-17T07:21:11.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:10 smithi044 ceph-mon[58191]: osdmap e78: 8 total, 7 up, 8 in 2024-09-17T07:21:11.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:10 smithi044 ceph-mon[58191]: from='client.44400 172.21.15.44:0/1285485983' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:21:11.075 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:10 smithi044 podman[64537]: 2024-09-17 07:21:10.680117855 +0000 UTC m=+0.016669349 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:21:11.075 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:10 smithi044 podman[64537]: 2024-09-17 07:21:10.795936428 +0000 UTC m=+0.132487906 container create 551a45914a79b1b42e3b09b6e97f3dccc514bdf62cb05dc95a23c5c453e001e0 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T07:21:11.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:10 smithi044 ceph-mon[60356]: osdmap e78: 8 total, 7 up, 8 in 2024-09-17T07:21:11.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:10 smithi044 ceph-mon[60356]: from='client.44400 172.21.15.44:0/1285485983' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:21:11.363 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:10 smithi179 ceph-mon[50085]: osdmap e78: 8 total, 7 up, 8 in 2024-09-17T07:21:11.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:10 smithi179 ceph-mon[50085]: from='client.44400 172.21.15.44:0/1285485983' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:21:11.561 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:11 smithi044 podman[64537]: 2024-09-17 07:21:11.090834297 +0000 UTC m=+0.427385784 container init 551a45914a79b1b42e3b09b6e97f3dccc514bdf62cb05dc95a23c5c453e001e0 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-17T07:21:11.561 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:11 smithi044 podman[64537]: 2024-09-17 07:21:11.095600802 +0000 UTC m=+0.432152284 container start 551a45914a79b1b42e3b09b6e97f3dccc514bdf62cb05dc95a23c5c453e001e0 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-17T07:21:11.561 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:11 smithi044 podman[64537]: 2024-09-17 07:21:11.174162723 +0000 UTC m=+0.510714219 container attach 551a45914a79b1b42e3b09b6e97f3dccc514bdf62cb05dc95a23c5c453e001e0 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0-deactivate, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-17T07:21:12.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:11 smithi044 ceph-mon[58191]: pgmap v49: 161 pgs: 21 stale+active+clean, 140 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:21:12.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:11 smithi044 ceph-mon[58191]: from='client.54289 172.21.15.44:0/4258761098' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:21:12.225 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:11 smithi044 ceph-mon[60356]: pgmap v49: 161 pgs: 21 stale+active+clean, 140 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:21:12.225 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:11 smithi044 ceph-mon[60356]: from='client.54289 172.21.15.44:0/4258761098' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:21:12.226 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:11 smithi044 podman[64537]: 2024-09-17 07:21:11.940290196 +0000 UTC m=+1.276841674 container died 551a45914a79b1b42e3b09b6e97f3dccc514bdf62cb05dc95a23c5c453e001e0 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2) 2024-09-17T07:21:12.363 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:11 smithi179 ceph-mon[50085]: pgmap v49: 161 pgs: 21 stale+active+clean, 140 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:21:12.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:11 smithi179 ceph-mon[50085]: from='client.54289 172.21.15.44:0/4258761098' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:21:13.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:12 smithi044 ceph-mon[58191]: pgmap v50: 161 pgs: 39 active+undersized, 19 active+undersized+degraded, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T07:21:13.225 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:12 smithi044 ceph-mon[58191]: Health check failed: Degraded data redundancy: 115/999 objects degraded (11.512%), 19 pgs degraded (PG_DEGRADED) 2024-09-17T07:21:13.225 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:12 smithi044 ceph-mon[60356]: pgmap v50: 161 pgs: 39 active+undersized, 19 active+undersized+degraded, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T07:21:13.225 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:12 smithi044 ceph-mon[60356]: Health check failed: Degraded data redundancy: 115/999 objects degraded (11.512%), 19 pgs degraded (PG_DEGRADED) 2024-09-17T07:21:13.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:12 smithi179 ceph-mon[50085]: pgmap v50: 161 pgs: 39 active+undersized, 19 active+undersized+degraded, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T07:21:13.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:12 smithi179 ceph-mon[50085]: Health check failed: Degraded data redundancy: 115/999 objects degraded (11.512%), 19 pgs degraded (PG_DEGRADED) 2024-09-17T07:21:13.574 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:13 smithi044 podman[64594]: 2024-09-17 07:21:13.362096566 +0000 UTC m=+2.041983164 container remove 551a45914a79b1b42e3b09b6e97f3dccc514bdf62cb05dc95a23c5c453e001e0 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2) 2024-09-17T07:21:13.574 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:13 smithi044 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.0.service: Deactivated successfully. 2024-09-17T07:21:13.575 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:13 smithi044 systemd[1]: Stopped Ceph osd.0 for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:21:13.575 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:13 smithi044 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.0.service: Consumed 1.202s CPU time. 2024-09-17T07:21:13.979 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:13 smithi044 systemd[1]: Starting Ceph osd.0 for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:21:13.979 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:13 smithi044 podman[64726]: 2024-09-17 07:21:13.737736882 +0000 UTC m=+0.016459869 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:21:13.979 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:13 smithi044 podman[64726]: 2024-09-17 07:21:13.855386815 +0000 UTC m=+0.134109829 container create f985374358aeb0a85a0a994d35405d8892a73fa92dfe30ad86d033026e8ff233 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0-activate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-17T07:21:14.324 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:14 smithi044 podman[64726]: 2024-09-17 07:21:14.09135721 +0000 UTC m=+0.370080191 container init f985374358aeb0a85a0a994d35405d8892a73fa92dfe30ad86d033026e8ff233 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0-activate, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-17T07:21:14.325 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:14 smithi044 podman[64726]: 2024-09-17 07:21:14.095035297 +0000 UTC m=+0.373758270 container start f985374358aeb0a85a0a994d35405d8892a73fa92dfe30ad86d033026e8ff233 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0-activate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b) 2024-09-17T07:21:14.325 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:14 smithi044 podman[64726]: 2024-09-17 07:21:14.183065951 +0000 UTC m=+0.461788938 container attach f985374358aeb0a85a0a994d35405d8892a73fa92dfe30ad86d033026e8ff233 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0-activate, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-17T07:21:14.825 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0-activate[64742]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-17T07:21:14.825 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:14 smithi044 bash[64726]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-17T07:21:14.825 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0-activate[64742]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-17T07:21:14.825 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:14 smithi044 bash[64726]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-17T07:21:14.825 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0-activate[64742]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-17T07:21:14.825 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:14 smithi044 bash[64726]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-17T07:21:14.825 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0-activate[64742]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-17T07:21:14.826 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:14 smithi044 bash[64726]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-17T07:21:14.826 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0-activate[64742]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-17T07:21:14.826 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:14 smithi044 bash[64726]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-17T07:21:14.826 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0-activate[64742]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-17T07:21:14.826 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:14 smithi044 bash[64726]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-17T07:21:14.826 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:14 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0-activate[64742]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-17T07:21:14.826 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:14 smithi044 bash[64726]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-17T07:21:14.826 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:14 smithi044 podman[64726]: 2024-09-17 07:21:14.450526601 +0000 UTC m=+0.729249581 container died f985374358aeb0a85a0a994d35405d8892a73fa92dfe30ad86d033026e8ff233 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0-activate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, ceph=True, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-17T07:21:15.574 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:15 smithi044 ceph-mon[60356]: pgmap v51: 161 pgs: 39 active+undersized, 19 active+undersized+degraded, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T07:21:15.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:15 smithi044 ceph-mon[58191]: pgmap v51: 161 pgs: 39 active+undersized, 19 active+undersized+degraded, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T07:21:15.575 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:15 smithi044 podman[64862]: 2024-09-17 07:21:15.237388281 +0000 UTC m=+0.780959148 container remove f985374358aeb0a85a0a994d35405d8892a73fa92dfe30ad86d033026e8ff233 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0-activate, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b) 2024-09-17T07:21:15.575 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:15 smithi044 podman[64921]: 2024-09-17 07:21:15.358198037 +0000 UTC m=+0.016974439 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:21:15.575 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:15 smithi044 podman[64921]: 2024-09-17 07:21:15.480628811 +0000 UTC m=+0.139405245 container create 3ba47a4e2e95e0cfb10d02cc4b46b7ed73dfb0a322d846624c5e9ac410a83214 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, ceph=True) 2024-09-17T07:21:15.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:15 smithi179 ceph-mon[50085]: pgmap v51: 161 pgs: 39 active+undersized, 19 active+undersized+degraded, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T07:21:16.075 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:15 smithi044 podman[64921]: 2024-09-17 07:21:15.699401748 +0000 UTC m=+0.358178146 container init 3ba47a4e2e95e0cfb10d02cc4b46b7ed73dfb0a322d846624c5e9ac410a83214 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, ceph=True) 2024-09-17T07:21:16.075 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:15 smithi044 podman[64921]: 2024-09-17 07:21:15.702737057 +0000 UTC m=+0.361513452 container start 3ba47a4e2e95e0cfb10d02cc4b46b7ed73dfb0a322d846624c5e9ac410a83214 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-17T07:21:16.075 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:15 smithi044 ceph-osd[64940]: -- 172.21.15.44:0/2619588224 <== mon.1 v2:172.21.15.44:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c7260a51e0 con 0x55c726e24000 2024-09-17T07:21:16.075 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:15 smithi044 bash[64921]: 3ba47a4e2e95e0cfb10d02cc4b46b7ed73dfb0a322d846624c5e9ac410a83214 2024-09-17T07:21:16.075 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:15 smithi044 systemd[1]: Started Ceph osd.0 for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:21:16.569 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:16 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0[64936]: 2024-09-17T07:21:16.242+0000 7f59d0a27640 -1 Falling back to public interface 2024-09-17T07:21:17.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:16 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:17.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:16 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:17.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:16 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:21:17.114 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:16 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Scheduled restart job, restart counter is at 10. 2024-09-17T07:21:17.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:16 smithi179 systemd[1]: Stopped Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:21:17.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:16 smithi179 systemd[1]: Starting Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:21:17.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 podman[52589]: 2024-09-17 07:21:17.044112755 +0000 UTC m=+0.107289092 container create f8557b2318b089ff1b0936e2b5b9a603bc0b1cbae96784c243bc947ce7644a17 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:21:17.115 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 podman[52589]: 2024-09-17 07:21:16.955207149 +0000 UTC m=+0.018383456 image pull de242295e2257c37c8cadfd962369228f8f10b2d48a44259b65fef44ad4f6490 docker.io/prom/prometheus:v2.18.1 2024-09-17T07:21:17.178 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:16 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:17.178 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:16 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:17.178 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:16 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:21:17.179 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:16 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:17.179 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:16 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:17.179 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:16 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:21:17.614 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 podman[52589]: 2024-09-17 07:21:17.262309834 +0000 UTC m=+0.325486126 container init f8557b2318b089ff1b0936e2b5b9a603bc0b1cbae96784c243bc947ce7644a17 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:21:17.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 podman[52589]: 2024-09-17 07:21:17.265117761 +0000 UTC m=+0.328294056 container start f8557b2318b089ff1b0936e2b5b9a603bc0b1cbae96784c243bc947ce7644a17 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:21:17.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.294Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-17T07:21:17.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.294Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2024-09-17T07:21:17.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.294Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2024-09-17T07:21:17.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.294Z caller=main.go:339 host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi179 (none))" 2024-09-17T07:21:17.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.294Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T07:21:17.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.294Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T07:21:17.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.295Z caller=main.go:678 msg="Starting TSDB ..." 2024-09-17T07:21:17.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.296Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2024-09-17T07:21:17.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.298Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2024-09-17T07:21:17.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.298Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=17 2024-09-17T07:21:17.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.304Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=17 2024-09-17T07:21:17.615 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.305Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=17 2024-09-17T07:21:17.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.305Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=17 2024-09-17T07:21:17.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.305Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=17 2024-09-17T07:21:17.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.305Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=17 2024-09-17T07:21:17.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.305Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=17 2024-09-17T07:21:17.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.305Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=7 maxSegment=17 2024-09-17T07:21:17.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.305Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=8 maxSegment=17 2024-09-17T07:21:17.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.305Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=9 maxSegment=17 2024-09-17T07:21:17.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.306Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=10 maxSegment=17 2024-09-17T07:21:17.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.306Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=11 maxSegment=17 2024-09-17T07:21:17.616 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.308Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=12 maxSegment=17 2024-09-17T07:21:17.617 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.308Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=13 maxSegment=17 2024-09-17T07:21:17.617 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.308Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=14 maxSegment=17 2024-09-17T07:21:17.617 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.308Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=15 maxSegment=17 2024-09-17T07:21:17.617 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.308Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=16 maxSegment=17 2024-09-17T07:21:17.617 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.308Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=17 maxSegment=17 2024-09-17T07:21:17.617 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.308Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=10.709379ms 2024-09-17T07:21:17.617 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.310Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2024-09-17T07:21:17.617 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.310Z caller=main.go:695 msg="TSDB started" 2024-09-17T07:21:17.617 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.310Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T07:21:17.617 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.310Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2024-09-17T07:21:17.617 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.310Z caller=main.go:561 msg="Stopping notify discovery manager..." 2024-09-17T07:21:17.617 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.310Z caller=main.go:583 msg="Stopping scrape manager..." 2024-09-17T07:21:17.617 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.310Z caller=main.go:557 msg="Notify discovery manager stopped" 2024-09-17T07:21:17.618 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.310Z caller=main.go:543 msg="Scrape discovery manager stopped" 2024-09-17T07:21:17.618 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.310Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2024-09-17T07:21:17.618 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.310Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2024-09-17T07:21:17.618 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.310Z caller=main.go:577 msg="Scrape manager stopped" 2024-09-17T07:21:17.618 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.311Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2024-09-17T07:21:17.618 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=info ts=2024-09-17T07:21:17.311Z caller=main.go:749 msg="Notifier manager stopped" 2024-09-17T07:21:17.618 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a[52604]: level=error ts=2024-09-17T07:21:17.311Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 14: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain\n line 31: field http_sd_configs not found in type config.plain\n line 45: field http_sd_configs not found in type config.plain\n line 49: field http_sd_configs not found in type config.plain" 2024-09-17T07:21:17.618 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 bash[52589]: f8557b2318b089ff1b0936e2b5b9a603bc0b1cbae96784c243bc947ce7644a17 2024-09-17T07:21:17.618 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 systemd[1]: Started Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:21:17.618 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 podman[52621]: 2024-09-17 07:21:17.353345749 +0000 UTC m=+0.035151028 container died f8557b2318b089ff1b0936e2b5b9a603bc0b1cbae96784c243bc947ce7644a17 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:21:17.907 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:17 smithi179 ceph-mon[50085]: pgmap v52: 161 pgs: 39 active+undersized, 19 active+undersized+degraded, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T07:21:17.908 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 podman[52621]: 2024-09-17 07:21:17.907890049 +0000 UTC m=+0.589695377 container remove f8557b2318b089ff1b0936e2b5b9a603bc0b1cbae96784c243bc947ce7644a17 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T07:21:18.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:17 smithi044 ceph-mon[58191]: pgmap v52: 161 pgs: 39 active+undersized, 19 active+undersized+degraded, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T07:21:18.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:17 smithi044 ceph-mon[60356]: pgmap v52: 161 pgs: 39 active+undersized, 19 active+undersized+degraded, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T07:21:18.364 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2024-09-17T07:21:18.364 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:17 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Failed with result 'exit-code'. 2024-09-17T07:21:19.242 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0[64936]: 2024-09-17T07:21:18.963+0000 7f59d0a27640 -1 osd.0 0 read_superblock omap replica is missing. 2024-09-17T07:21:19.242 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:18 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0[64936]: 2024-09-17T07:21:18.991+0000 7f59d0a27640 -1 osd.0 76 log_to_monitors true 2024-09-17T07:21:20.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:19 smithi179 ceph-mon[50085]: pgmap v53: 161 pgs: 39 active+undersized, 19 active+undersized+degraded, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T07:21:20.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:19 smithi179 ceph-mon[50085]: from='osd.0 [v2:172.21.15.44:6802/1556061536,v1:172.21.15.44:6803/1556061536]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T07:21:20.323 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:19 smithi044 ceph-mon[58191]: pgmap v53: 161 pgs: 39 active+undersized, 19 active+undersized+degraded, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T07:21:20.323 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:19 smithi044 ceph-mon[58191]: from='osd.0 [v2:172.21.15.44:6802/1556061536,v1:172.21.15.44:6803/1556061536]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T07:21:20.323 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:19 smithi044 ceph-mon[60356]: pgmap v53: 161 pgs: 39 active+undersized, 19 active+undersized+degraded, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T07:21:20.323 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:19 smithi044 ceph-mon[60356]: from='osd.0 [v2:172.21.15.44:6802/1556061536,v1:172.21.15.44:6803/1556061536]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T07:21:20.574 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:21:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0[64936]: 2024-09-17T07:21:20.321+0000 7f59c47c9640 -1 osd.0 76 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T07:21:21.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:20 smithi179 ceph-mon[50085]: from='osd.0 [v2:172.21.15.44:6802/1556061536,v1:172.21.15.44:6803/1556061536]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-17T07:21:21.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:20 smithi179 ceph-mon[50085]: osdmap e79: 8 total, 7 up, 8 in 2024-09-17T07:21:21.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:20 smithi179 ceph-mon[50085]: from='osd.0 [v2:172.21.15.44:6802/1556061536,v1:172.21.15.44:6803/1556061536]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-09-17T07:21:21.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:20 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:21.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:20 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:21.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:20 smithi179 ceph-mon[50085]: from='osd.0 [v2:172.21.15.44:6802/1556061536,v1:172.21.15.44:6803/1556061536]' entity='osd.0' 2024-09-17T07:21:21.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:20 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:21.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:20 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:21.195 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:20 smithi044 ceph-mon[58191]: from='osd.0 [v2:172.21.15.44:6802/1556061536,v1:172.21.15.44:6803/1556061536]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-17T07:21:21.195 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:20 smithi044 ceph-mon[58191]: osdmap e79: 8 total, 7 up, 8 in 2024-09-17T07:21:21.195 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:20 smithi044 ceph-mon[58191]: from='osd.0 [v2:172.21.15.44:6802/1556061536,v1:172.21.15.44:6803/1556061536]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-09-17T07:21:21.195 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:20 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:21.195 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:20 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:21.195 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:20 smithi044 ceph-mon[58191]: from='osd.0 [v2:172.21.15.44:6802/1556061536,v1:172.21.15.44:6803/1556061536]' entity='osd.0' 2024-09-17T07:21:21.195 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:20 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:21.195 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:20 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:21.196 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:20 smithi044 ceph-mon[60356]: from='osd.0 [v2:172.21.15.44:6802/1556061536,v1:172.21.15.44:6803/1556061536]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-17T07:21:21.196 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:20 smithi044 ceph-mon[60356]: osdmap e79: 8 total, 7 up, 8 in 2024-09-17T07:21:21.196 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:20 smithi044 ceph-mon[60356]: from='osd.0 [v2:172.21.15.44:6802/1556061536,v1:172.21.15.44:6803/1556061536]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-09-17T07:21:21.196 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:20 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:21.196 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:20 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:21.196 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:20 smithi044 ceph-mon[60356]: from='osd.0 [v2:172.21.15.44:6802/1556061536,v1:172.21.15.44:6803/1556061536]' entity='osd.0' 2024-09-17T07:21:21.196 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:20 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:21.196 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:20 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:22.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:21 smithi179 ceph-mon[50085]: pgmap v55: 161 pgs: 39 active+undersized, 19 active+undersized+degraded, 103 active+clean; 604 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T07:21:22.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:21 smithi179 ceph-mon[50085]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T07:21:22.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:21 smithi179 ceph-mon[50085]: osd.0 [v2:172.21.15.44:6802/1556061536,v1:172.21.15.44:6803/1556061536] boot 2024-09-17T07:21:22.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:21 smithi179 ceph-mon[50085]: osdmap e80: 8 total, 8 up, 8 in 2024-09-17T07:21:22.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:21 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T07:21:22.227 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:21 smithi044 ceph-mon[58191]: pgmap v55: 161 pgs: 39 active+undersized, 19 active+undersized+degraded, 103 active+clean; 604 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T07:21:22.227 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:21 smithi044 ceph-mon[58191]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T07:21:22.227 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:21 smithi044 ceph-mon[58191]: osd.0 [v2:172.21.15.44:6802/1556061536,v1:172.21.15.44:6803/1556061536] boot 2024-09-17T07:21:22.227 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:21 smithi044 ceph-mon[58191]: osdmap e80: 8 total, 8 up, 8 in 2024-09-17T07:21:22.228 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:21 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T07:21:22.228 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:21 smithi044 ceph-mon[60356]: pgmap v55: 161 pgs: 39 active+undersized, 19 active+undersized+degraded, 103 active+clean; 604 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T07:21:22.228 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:21 smithi044 ceph-mon[60356]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T07:21:22.228 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:21 smithi044 ceph-mon[60356]: osd.0 [v2:172.21.15.44:6802/1556061536,v1:172.21.15.44:6803/1556061536] boot 2024-09-17T07:21:22.228 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:21 smithi044 ceph-mon[60356]: osdmap e80: 8 total, 8 up, 8 in 2024-09-17T07:21:22.228 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:21 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T07:21:23.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:23 smithi179 ceph-mon[50085]: pgmap v57: 161 pgs: 39 active+undersized, 19 active+undersized+degraded, 103 active+clean; 604 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T07:21:23.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:23 smithi179 ceph-mon[50085]: osdmap e81: 8 total, 8 up, 8 in 2024-09-17T07:21:23.656 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:23 smithi044 ceph-mon[58191]: pgmap v57: 161 pgs: 39 active+undersized, 19 active+undersized+degraded, 103 active+clean; 604 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T07:21:23.657 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:23 smithi044 ceph-mon[58191]: osdmap e81: 8 total, 8 up, 8 in 2024-09-17T07:21:23.657 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:23 smithi044 ceph-mon[60356]: pgmap v57: 161 pgs: 39 active+undersized, 19 active+undersized+degraded, 103 active+clean; 604 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T07:21:23.657 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:23 smithi044 ceph-mon[60356]: osdmap e81: 8 total, 8 up, 8 in 2024-09-17T07:21:25.282 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:21:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:21:25.182+0000 7f5a7ae47640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-09-17T07:21:25.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:25 smithi044 ceph-mon[60356]: pgmap v59: 161 pgs: 39 active+undersized, 19 active+undersized+degraded, 103 active+clean; 604 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T07:21:25.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:25 smithi044 ceph-mon[60356]: from='client.54313 172.21.15.44:0/4103697885' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:21:25.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:25 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:25.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:25 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:25.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:25 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:21:25.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:25 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:21:25.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:25 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:25.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:25 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:21:25.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:25 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:21:25.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:25 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:21:25.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:25 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:21:25.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:25 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T07:21:25.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:25 smithi044 ceph-mon[60356]: from='client.54319 172.21.15.44:0/956762568' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:21:25.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:25 smithi044 ceph-mon[58191]: pgmap v59: 161 pgs: 39 active+undersized, 19 active+undersized+degraded, 103 active+clean; 604 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T07:21:25.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:25 smithi044 ceph-mon[58191]: from='client.54313 172.21.15.44:0/4103697885' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:21:25.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:25 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:25.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:25 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:25.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:25 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:21:25.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:25 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:21:25.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:25 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:25.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:25 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:21:25.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:25 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:21:25.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:25 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:21:25.577 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:25 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:21:25.577 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:25 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T07:21:25.577 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:25 smithi044 ceph-mon[58191]: from='client.54319 172.21.15.44:0/956762568' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:21:25.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:25 smithi179 ceph-mon[50085]: pgmap v59: 161 pgs: 39 active+undersized, 19 active+undersized+degraded, 103 active+clean; 604 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T07:21:25.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:25 smithi179 ceph-mon[50085]: from='client.54313 172.21.15.44:0/4103697885' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:21:25.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:25 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:25.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:25 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:25.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:25 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:21:25.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:25 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:21:25.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:25 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:25.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:25 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:21:25.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:25 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:21:25.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:25 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:21:25.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:25 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:21:25.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:25 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T07:21:25.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:25 smithi179 ceph-mon[50085]: from='client.54319 172.21.15.44:0/956762568' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:21:26.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:26 smithi179 ceph-mon[50085]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:21:26.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:26 smithi179 ceph-mon[50085]: pgmap v60: 161 pgs: 26 active+undersized, 11 active+undersized+degraded, 124 active+clean; 604 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 62/999 objects degraded (6.206%) 2024-09-17T07:21:26.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:26 smithi179 ceph-mon[50085]: pgmap v61: 161 pgs: 26 active+undersized, 11 active+undersized+degraded, 124 active+clean; 604 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 62/999 objects degraded (6.206%) 2024-09-17T07:21:26.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:26 smithi179 ceph-mon[50085]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T07:21:26.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:26 smithi179 ceph-mon[50085]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-09-17T07:21:26.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:26 smithi179 ceph-mon[50085]: Health check update: Degraded data redundancy: 62/999 objects degraded (6.206%), 11 pgs degraded (PG_DEGRADED) 2024-09-17T07:21:26.760 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:26 smithi044 ceph-mon[58191]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:21:26.760 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:26 smithi044 ceph-mon[58191]: pgmap v60: 161 pgs: 26 active+undersized, 11 active+undersized+degraded, 124 active+clean; 604 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 62/999 objects degraded (6.206%) 2024-09-17T07:21:26.760 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:26 smithi044 ceph-mon[58191]: pgmap v61: 161 pgs: 26 active+undersized, 11 active+undersized+degraded, 124 active+clean; 604 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 62/999 objects degraded (6.206%) 2024-09-17T07:21:26.760 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:26 smithi044 ceph-mon[58191]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T07:21:26.760 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:26 smithi044 ceph-mon[58191]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-09-17T07:21:26.760 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:26 smithi044 ceph-mon[58191]: Health check update: Degraded data redundancy: 62/999 objects degraded (6.206%), 11 pgs degraded (PG_DEGRADED) 2024-09-17T07:21:26.760 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:26 smithi044 ceph-mon[60356]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:21:26.761 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:26 smithi044 ceph-mon[60356]: pgmap v60: 161 pgs: 26 active+undersized, 11 active+undersized+degraded, 124 active+clean; 604 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 62/999 objects degraded (6.206%) 2024-09-17T07:21:26.761 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:26 smithi044 ceph-mon[60356]: pgmap v61: 161 pgs: 26 active+undersized, 11 active+undersized+degraded, 124 active+clean; 604 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 62/999 objects degraded (6.206%) 2024-09-17T07:21:26.761 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:26 smithi044 ceph-mon[60356]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T07:21:26.761 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:26 smithi044 ceph-mon[60356]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-09-17T07:21:26.761 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:26 smithi044 ceph-mon[60356]: Health check update: Degraded data redundancy: 62/999 objects degraded (6.206%), 11 pgs degraded (PG_DEGRADED) 2024-09-17T07:21:27.653 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:27 smithi044 ceph-mon[58191]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 62/999 objects degraded (6.206%), 11 pgs degraded) 2024-09-17T07:21:27.653 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:27 smithi044 ceph-mon[60356]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 62/999 objects degraded (6.206%), 11 pgs degraded) 2024-09-17T07:21:27.789 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-09-17T07:21:27.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:27 smithi179 ceph-mon[50085]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 62/999 objects degraded (6.206%), 11 pgs degraded) 2024-09-17T07:21:28.397 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:28 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Scheduled restart job, restart counter is at 11. 2024-09-17T07:21:28.397 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:28 smithi179 systemd[1]: Stopped Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:21:28.398 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:28 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Start request repeated too quickly. 2024-09-17T07:21:28.398 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:28 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service: Failed with result 'exit-code'. 2024-09-17T07:21:28.398 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:21:28 smithi179 systemd[1]: Failed to start Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:21:28.455 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T07:21:28.455 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.a smithi044 *:9093,9094 running (3m) 8s ago 7m 18.9M - 0.20.0 0881eb8f169f 0473d4b023ad 2024-09-17T07:21:28.455 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.a smithi179 *:3000 running (2m) 55s ago 7m 27.2M - 6.7.4 557c83e11646 43ab0e03dcac 2024-09-17T07:21:28.455 INFO:teuthology.orchestra.run.smithi044.stdout:iscsi.foo.smithi044.elmtfk smithi044 unknown 8s ago 6m - - 2024-09-17T07:21:28.455 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.x smithi179 *:8443,8765 running (2m) 55s ago 11m 436M - 18.2.4-855-g19e2a2be 980a0bfcd684 66b52dc26022 2024-09-17T07:21:28.455 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.y smithi044 *:8443,8765 running (2m) 8s ago 13m 478M - 18.2.4-855-g19e2a2be 980a0bfcd684 8080dd318a1c 2024-09-17T07:21:28.455 INFO:teuthology.orchestra.run.smithi044.stdout:mon.a smithi044 running (105s) 8s ago 13m 43.4M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 c3293e8cdbba 2024-09-17T07:21:28.456 INFO:teuthology.orchestra.run.smithi044.stdout:mon.b smithi179 running (63s) 55s ago 12m 27.7M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 ade749759184 2024-09-17T07:21:28.456 INFO:teuthology.orchestra.run.smithi044.stdout:mon.c smithi044 running (82s) 8s ago 12m 37.6M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 02f85d31e0bd 2024-09-17T07:21:28.456 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.a smithi044 *:9100 running (3m) 8s ago 8m 17.1M - 0.18.1 e5a616e4b9cf 8a2cde5bd9ed 2024-09-17T07:21:28.456 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.b smithi179 *:9100 running (2m) 55s ago 8m 8929k - 0.18.1 e5a616e4b9cf 04c63b5cb524 2024-09-17T07:21:28.456 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (13s) 8s ago 11m 11.8M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 3ba47a4e2e95 2024-09-17T07:21:28.456 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (11m) 8s ago 11m 47.0M 4096M 16.2.0 24ecd6d5f14c 2d5122cbeff9 2024-09-17T07:21:28.456 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (10m) 8s ago 10m 44.3M 4096M 16.2.0 24ecd6d5f14c 7fd2f30be874 2024-09-17T07:21:28.456 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi044 running (10m) 8s ago 10m 48.7M 4096M 16.2.0 24ecd6d5f14c 2b7d623395a7 2024-09-17T07:21:28.456 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi179 running (10m) 55s ago 10m 51.7M 4096M 16.2.0 24ecd6d5f14c 911e6f47c822 2024-09-17T07:21:28.456 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi179 running (10m) 55s ago 10m 47.8M 4096M 16.2.0 24ecd6d5f14c e56060742dff 2024-09-17T07:21:28.456 INFO:teuthology.orchestra.run.smithi044.stdout:osd.6 smithi179 running (9m) 55s ago 9m 47.4M 4096M 16.2.0 24ecd6d5f14c 7a5c30b68cbc 2024-09-17T07:21:28.456 INFO:teuthology.orchestra.run.smithi044.stdout:osd.7 smithi179 running (9m) 55s ago 9m 53.5M 4096M 16.2.0 24ecd6d5f14c b0d186fbf8a6 2024-09-17T07:21:28.456 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.a smithi179 *:9095 unknown 55s ago 8m - - 2024-09-17T07:21:28.456 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi044.nhyhnh smithi044 *:8000 running (7m) 8s ago 7m 38.7M - 16.2.0 24ecd6d5f14c 9b455479614f 2024-09-17T07:21:28.457 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi179.tqlzyy smithi179 *:8000 running (7m) 55s ago 7m 41.2M - 16.2.0 24ecd6d5f14c c8d1693241ce 2024-09-17T07:21:28.457 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi044.vnufiz smithi044 *:80 running (7m) 8s ago 7m 38.8M - 16.2.0 24ecd6d5f14c 6eae2bff5ec8 2024-09-17T07:21:28.457 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi179.ecjhll smithi179 *:80 running (7m) 55s ago 7m 41.7M - 16.2.0 24ecd6d5f14c 253109ee0394 2024-09-17T07:21:28.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:28 smithi044 ceph-mon[58191]: pgmap v62: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 179 B/s rd, 0 op/s 2024-09-17T07:21:28.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:28 smithi044 ceph-mon[58191]: from='client.44442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:21:28.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:28 smithi044 ceph-mon[60356]: pgmap v62: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 179 B/s rd, 0 op/s 2024-09-17T07:21:28.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:28 smithi044 ceph-mon[60356]: from='client.44442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:21:28.858 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:21:28.858 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-09-17T07:21:28.858 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3 2024-09-17T07:21:28.858 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:21:28.858 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-09-17T07:21:28.858 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T07:21:28.858 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:21:28.858 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-09-17T07:21:28.858 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 7, 2024-09-17T07:21:28.859 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 1 2024-09-17T07:21:28.859 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:21:28.859 INFO:teuthology.orchestra.run.smithi044.stdout: "rgw": { 2024-09-17T07:21:28.859 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2024-09-17T07:21:28.859 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:21:28.859 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-09-17T07:21:28.859 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 11, 2024-09-17T07:21:28.859 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 6 2024-09-17T07:21:28.859 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-09-17T07:21:28.859 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:21:28.863 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:28 smithi179 ceph-mon[50085]: pgmap v62: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 179 B/s rd, 0 op/s 2024-09-17T07:21:28.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:28 smithi179 ceph-mon[50085]: from='client.44442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:21:29.198 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:21:29.198 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9", 2024-09-17T07:21:29.198 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-09-17T07:21:29.198 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T07:21:29.199 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [ 2024-09-17T07:21:29.199 INFO:teuthology.orchestra.run.smithi044.stdout: "mon", 2024-09-17T07:21:29.199 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr" 2024-09-17T07:21:29.199 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-09-17T07:21:29.199 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "6/23 daemons upgraded", 2024-09-17T07:21:29.199 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "Currently upgrading osd daemons", 2024-09-17T07:21:29.199 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-09-17T07:21:29.199 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:21:29.574 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:29 smithi044 ceph-mon[60356]: from='client.54331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:21:29.574 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:29 smithi044 ceph-mon[60356]: from='client.44454 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:21:29.574 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:29 smithi044 ceph-mon[60356]: from='client.44460 172.21.15.44:0/2941280718' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:21:29.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:29 smithi044 ceph-mon[58191]: from='client.54331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:21:29.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:29 smithi044 ceph-mon[58191]: from='client.44454 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:21:29.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:29 smithi044 ceph-mon[58191]: from='client.44460 172.21.15.44:0/2941280718' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:21:29.597 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_WARN 2 failed cephadm daemon(s) 2024-09-17T07:21:29.597 INFO:teuthology.orchestra.run.smithi044.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T07:21:29.597 INFO:teuthology.orchestra.run.smithi044.stdout: daemon iscsi.foo.smithi044.elmtfk on smithi044 is in unknown state 2024-09-17T07:21:29.597 INFO:teuthology.orchestra.run.smithi044.stdout: daemon prometheus.a on smithi179 is in unknown state 2024-09-17T07:21:29.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:29 smithi179 ceph-mon[50085]: from='client.54331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:21:29.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:29 smithi179 ceph-mon[50085]: from='client.44454 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:21:29.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:29 smithi179 ceph-mon[50085]: from='client.44460 172.21.15.44:0/2941280718' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:21:30.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:30 smithi044 ceph-mon[58191]: pgmap v63: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 152 B/s rd, 0 op/s 2024-09-17T07:21:30.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:30 smithi044 ceph-mon[58191]: from='client.54349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:21:30.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:30 smithi044 ceph-mon[58191]: from='client.54355 172.21.15.44:0/252724771' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:21:30.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:30 smithi044 ceph-mon[60356]: pgmap v63: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 152 B/s rd, 0 op/s 2024-09-17T07:21:30.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:30 smithi044 ceph-mon[60356]: from='client.54349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:21:30.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:30 smithi044 ceph-mon[60356]: from='client.54355 172.21.15.44:0/252724771' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:21:30.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:30 smithi179 ceph-mon[50085]: pgmap v63: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 152 B/s rd, 0 op/s 2024-09-17T07:21:30.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:30 smithi179 ceph-mon[50085]: from='client.54349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:21:30.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:30 smithi179 ceph-mon[50085]: from='client.54355 172.21.15.44:0/252724771' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:21:32.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:32 smithi044 ceph-mon[58191]: pgmap v64: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 117 B/s rd, 0 op/s 2024-09-17T07:21:32.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:32 smithi044 ceph-mon[60356]: pgmap v64: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 117 B/s rd, 0 op/s 2024-09-17T07:21:32.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:32 smithi179 ceph-mon[50085]: pgmap v64: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 117 B/s rd, 0 op/s 2024-09-17T07:21:34.666 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:34 smithi179 ceph-mon[50085]: pgmap v65: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 117 B/s rd, 0 op/s 2024-09-17T07:21:34.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:34 smithi044 ceph-mon[58191]: pgmap v65: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 117 B/s rd, 0 op/s 2024-09-17T07:21:34.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:34 smithi044 ceph-mon[60356]: pgmap v65: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 117 B/s rd, 0 op/s 2024-09-17T07:21:36.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:36 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:36.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:36 smithi179 ceph-mon[50085]: pgmap v66: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-09-17T07:21:36.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:36 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:36.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:36 smithi044 ceph-mon[58191]: pgmap v66: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-09-17T07:21:36.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:36 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:36.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:36 smithi044 ceph-mon[60356]: pgmap v66: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-09-17T07:21:38.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:38 smithi179 ceph-mon[50085]: pgmap v67: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:21:38.558 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:38 smithi044 ceph-mon[58191]: pgmap v67: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:21:38.559 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:38 smithi044 ceph-mon[60356]: pgmap v67: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:21:39.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:39 smithi044 ceph-mon[58191]: from='client.54373 172.21.15.44:0/2553490731' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:21:39.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:39 smithi044 ceph-mon[58191]: from='client.54379 172.21.15.44:0/1405532311' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:21:39.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:39 smithi044 ceph-mon[60356]: from='client.54373 172.21.15.44:0/2553490731' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:21:39.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:39 smithi044 ceph-mon[60356]: from='client.54379 172.21.15.44:0/1405532311' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:21:39.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:39 smithi179 ceph-mon[50085]: from='client.54373 172.21.15.44:0/2553490731' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:21:39.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:39 smithi179 ceph-mon[50085]: from='client.54379 172.21.15.44:0/1405532311' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:21:40.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:40 smithi179 ceph-mon[50085]: pgmap v68: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:21:40.547 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:40 smithi044 ceph-mon[58191]: pgmap v68: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:21:40.547 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:40 smithi044 ceph-mon[60356]: pgmap v68: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:21:41.508 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:41 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T07:21:41.509 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:41 smithi044 ceph-mon[60356]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T07:21:41.509 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:41 smithi044 ceph-mon[60356]: Upgrade: osd.1 is safe to restart 2024-09-17T07:21:41.509 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:41 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T07:21:41.509 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:41 smithi044 ceph-mon[58191]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T07:21:41.509 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:41 smithi044 ceph-mon[58191]: Upgrade: osd.1 is safe to restart 2024-09-17T07:21:41.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:41 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T07:21:41.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:41 smithi179 ceph-mon[50085]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T07:21:41.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:41 smithi179 ceph-mon[50085]: Upgrade: osd.1 is safe to restart 2024-09-17T07:21:42.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:42 smithi044 ceph-mon[58191]: pgmap v69: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:21:42.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:42 smithi044 ceph-mon[58191]: Upgrade: Updating osd.1 2024-09-17T07:21:42.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:42 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:42.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:42 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-17T07:21:42.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:42 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:21:42.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:42 smithi044 ceph-mon[58191]: Deploying daemon osd.1 on smithi044 2024-09-17T07:21:42.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:42 smithi044 ceph-mon[60356]: pgmap v69: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:21:42.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:42 smithi044 ceph-mon[60356]: Upgrade: Updating osd.1 2024-09-17T07:21:42.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:42 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:42.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:42 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-17T07:21:42.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:42 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:21:42.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:42 smithi044 ceph-mon[60356]: Deploying daemon osd.1 on smithi044 2024-09-17T07:21:42.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:42 smithi179 ceph-mon[50085]: pgmap v69: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:21:42.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:42 smithi179 ceph-mon[50085]: Upgrade: Updating osd.1 2024-09-17T07:21:42.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:42 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:42.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:42 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-17T07:21:42.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:42 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:21:42.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:42 smithi179 ceph-mon[50085]: Deploying daemon osd.1 on smithi044 2024-09-17T07:21:44.795 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:44 smithi044 ceph-mon[58191]: pgmap v70: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:21:44.795 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:44 smithi044 ceph-mon[60356]: pgmap v70: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:21:44.795 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:44 smithi044 systemd[1]: Stopping Ceph osd.1 for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:21:44.796 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.1[31665]: debug 2024-09-17T07:21:44.769+0000 7f9562273700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-17T07:21:44.796 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.1[31665]: debug 2024-09-17T07:21:44.769+0000 7f9562273700 -1 osd.1 81 *** Got signal Terminated *** 2024-09-17T07:21:44.796 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:44 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.1[31665]: debug 2024-09-17T07:21:44.769+0000 7f9562273700 -1 osd.1 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T07:21:44.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:44 smithi179 ceph-mon[50085]: pgmap v70: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:21:45.074 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:44 smithi044 podman[67460]: 2024-09-17 07:21:44.801011613 +0000 UTC m=+0.128335827 container died 2d5122cbeff94c32bb837389ffee0bf1153fb7179bad09e1446caecce5b3e25e (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.1, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20201204, CEPH_POINT_RELEASE=-16.2.0, maintainer=Dimitri Savineau , org.label-schema.name=CentOS Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=HEAD) 2024-09-17T07:21:45.454 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:45 smithi044 podman[67460]: 2024-09-17 07:21:45.169574524 +0000 UTC m=+0.496898735 container cleanup 2d5122cbeff94c32bb837389ffee0bf1153fb7179bad09e1446caecce5b3e25e (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.1, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Dimitri Savineau , org.label-schema.schema-version=1.0, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, ceph=True, org.label-schema.name=CentOS Base Image, org.label-schema.vendor=CentOS, RELEASE=HEAD, org.label-schema.build-date=20201204, CEPH_POINT_RELEASE=-16.2.0) 2024-09-17T07:21:45.454 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:45 smithi044 bash[67460]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.1 2024-09-17T07:21:45.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 reported immediately failed by osd.6 2024-09-17T07:21:45.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 failed (root=default,host=smithi044) (connection refused reported by osd.6) 2024-09-17T07:21:45.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 reported immediately failed by osd.5 2024-09-17T07:21:45.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 reported immediately failed by osd.4 2024-09-17T07:21:45.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 reported immediately failed by osd.0 2024-09-17T07:21:45.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 reported immediately failed by osd.7 2024-09-17T07:21:45.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 reported immediately failed by osd.6 2024-09-17T07:21:45.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 reported immediately failed by osd.5 2024-09-17T07:21:45.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 reported immediately failed by osd.4 2024-09-17T07:21:45.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 reported immediately failed by osd.0 2024-09-17T07:21:45.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 reported immediately failed by osd.3 2024-09-17T07:21:45.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 reported immediately failed by osd.2 2024-09-17T07:21:45.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 reported immediately failed by osd.7 2024-09-17T07:21:45.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 reported immediately failed by osd.3 2024-09-17T07:21:45.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 reported immediately failed by osd.2 2024-09-17T07:21:45.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 reported immediately failed by osd.5 2024-09-17T07:21:45.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 reported immediately failed by osd.4 2024-09-17T07:21:45.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 reported immediately failed by osd.3 2024-09-17T07:21:45.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 reported immediately failed by osd.7 2024-09-17T07:21:45.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 reported immediately failed by osd.5 2024-09-17T07:21:45.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 reported immediately failed by osd.7 2024-09-17T07:21:45.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 reported immediately failed by osd.4 2024-09-17T07:21:45.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 reported immediately failed by osd.3 2024-09-17T07:21:45.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 reported immediately failed by osd.6 2024-09-17T07:21:45.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 reported immediately failed by osd.4 2024-09-17T07:21:45.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 reported immediately failed by osd.0 2024-09-17T07:21:45.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 reported immediately failed by osd.6 2024-09-17T07:21:45.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 reported immediately failed by osd.5 2024-09-17T07:21:45.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 reported immediately failed by osd.4 2024-09-17T07:21:45.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 reported immediately failed by osd.0 2024-09-17T07:21:45.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 reported immediately failed by osd.3 2024-09-17T07:21:45.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 reported immediately failed by osd.5 2024-09-17T07:21:45.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 reported immediately failed by osd.2 2024-09-17T07:21:45.617 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 reported immediately failed by osd.7 2024-09-17T07:21:45.617 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 reported immediately failed by osd.3 2024-09-17T07:21:45.617 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 reported immediately failed by osd.2 2024-09-17T07:21:45.617 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:45 smithi179 ceph-mon[50085]: osd.1 reported immediately failed by osd.7 2024-09-17T07:21:45.824 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:45 smithi044 podman[67475]: 2024-09-17 07:21:45.454608483 +0000 UTC m=+0.652684491 container remove 2d5122cbeff94c32bb837389ffee0bf1153fb7179bad09e1446caecce5b3e25e (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, ceph=True, org.label-schema.build-date=20201204, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image, CEPH_POINT_RELEASE=-16.2.0, RELEASE=HEAD, maintainer=Dimitri Savineau ) 2024-09-17T07:21:45.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 reported immediately failed by osd.6 2024-09-17T07:21:45.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 failed (root=default,host=smithi044) (connection refused reported by osd.6) 2024-09-17T07:21:45.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 reported immediately failed by osd.5 2024-09-17T07:21:45.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 reported immediately failed by osd.4 2024-09-17T07:21:45.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 reported immediately failed by osd.0 2024-09-17T07:21:45.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 reported immediately failed by osd.7 2024-09-17T07:21:45.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 reported immediately failed by osd.6 2024-09-17T07:21:45.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 reported immediately failed by osd.5 2024-09-17T07:21:45.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 reported immediately failed by osd.4 2024-09-17T07:21:45.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 reported immediately failed by osd.0 2024-09-17T07:21:45.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 reported immediately failed by osd.3 2024-09-17T07:21:45.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 reported immediately failed by osd.2 2024-09-17T07:21:45.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 reported immediately failed by osd.7 2024-09-17T07:21:45.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 reported immediately failed by osd.3 2024-09-17T07:21:45.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 reported immediately failed by osd.2 2024-09-17T07:21:45.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 reported immediately failed by osd.5 2024-09-17T07:21:45.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 reported immediately failed by osd.4 2024-09-17T07:21:45.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 reported immediately failed by osd.3 2024-09-17T07:21:45.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 reported immediately failed by osd.7 2024-09-17T07:21:45.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 reported immediately failed by osd.5 2024-09-17T07:21:45.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 reported immediately failed by osd.7 2024-09-17T07:21:45.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 reported immediately failed by osd.4 2024-09-17T07:21:45.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 reported immediately failed by osd.3 2024-09-17T07:21:45.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 reported immediately failed by osd.6 2024-09-17T07:21:45.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 reported immediately failed by osd.4 2024-09-17T07:21:45.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 reported immediately failed by osd.0 2024-09-17T07:21:45.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 reported immediately failed by osd.6 2024-09-17T07:21:45.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 reported immediately failed by osd.5 2024-09-17T07:21:45.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 reported immediately failed by osd.4 2024-09-17T07:21:45.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 reported immediately failed by osd.0 2024-09-17T07:21:45.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 reported immediately failed by osd.3 2024-09-17T07:21:45.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 reported immediately failed by osd.5 2024-09-17T07:21:45.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 reported immediately failed by osd.2 2024-09-17T07:21:45.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 reported immediately failed by osd.7 2024-09-17T07:21:45.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 reported immediately failed by osd.3 2024-09-17T07:21:45.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 reported immediately failed by osd.2 2024-09-17T07:21:45.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[60356]: osd.1 reported immediately failed by osd.7 2024-09-17T07:21:45.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 reported immediately failed by osd.6 2024-09-17T07:21:45.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 failed (root=default,host=smithi044) (connection refused reported by osd.6) 2024-09-17T07:21:45.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 reported immediately failed by osd.5 2024-09-17T07:21:45.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 reported immediately failed by osd.4 2024-09-17T07:21:45.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 reported immediately failed by osd.0 2024-09-17T07:21:45.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 reported immediately failed by osd.7 2024-09-17T07:21:45.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 reported immediately failed by osd.6 2024-09-17T07:21:45.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 reported immediately failed by osd.5 2024-09-17T07:21:45.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 reported immediately failed by osd.4 2024-09-17T07:21:45.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 reported immediately failed by osd.0 2024-09-17T07:21:45.828 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 reported immediately failed by osd.3 2024-09-17T07:21:45.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 reported immediately failed by osd.2 2024-09-17T07:21:45.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 reported immediately failed by osd.7 2024-09-17T07:21:45.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 reported immediately failed by osd.3 2024-09-17T07:21:45.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 reported immediately failed by osd.2 2024-09-17T07:21:45.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 reported immediately failed by osd.5 2024-09-17T07:21:45.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 reported immediately failed by osd.4 2024-09-17T07:21:45.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 reported immediately failed by osd.3 2024-09-17T07:21:45.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 reported immediately failed by osd.7 2024-09-17T07:21:45.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 reported immediately failed by osd.5 2024-09-17T07:21:45.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 reported immediately failed by osd.7 2024-09-17T07:21:45.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 reported immediately failed by osd.4 2024-09-17T07:21:45.829 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 reported immediately failed by osd.3 2024-09-17T07:21:45.830 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 reported immediately failed by osd.6 2024-09-17T07:21:45.830 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 reported immediately failed by osd.4 2024-09-17T07:21:45.830 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 reported immediately failed by osd.0 2024-09-17T07:21:45.830 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 reported immediately failed by osd.6 2024-09-17T07:21:45.830 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 reported immediately failed by osd.5 2024-09-17T07:21:45.830 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 reported immediately failed by osd.4 2024-09-17T07:21:45.830 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 reported immediately failed by osd.0 2024-09-17T07:21:45.830 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 reported immediately failed by osd.3 2024-09-17T07:21:45.830 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 reported immediately failed by osd.5 2024-09-17T07:21:45.830 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 reported immediately failed by osd.2 2024-09-17T07:21:45.830 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 reported immediately failed by osd.7 2024-09-17T07:21:45.830 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 reported immediately failed by osd.3 2024-09-17T07:21:45.830 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 reported immediately failed by osd.2 2024-09-17T07:21:45.830 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:45 smithi044 ceph-mon[58191]: osd.1 reported immediately failed by osd.7 2024-09-17T07:21:46.192 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:45 smithi044 podman[67578]: 2024-09-17 07:21:45.805542049 +0000 UTC m=+0.016665953 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:21:46.192 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:45 smithi044 podman[67578]: 2024-09-17 07:21:45.956226113 +0000 UTC m=+0.167350004 container create 9610329a93167ffd1303e701d7536eb80c10313867fc6763bbec90b14ddef0c5 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-17T07:21:46.192 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:46 smithi044 podman[67578]: 2024-09-17 07:21:46.192451094 +0000 UTC m=+0.403574996 container init 9610329a93167ffd1303e701d7536eb80c10313867fc6763bbec90b14ddef0c5 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b) 2024-09-17T07:21:46.544 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:46 smithi044 podman[67578]: 2024-09-17 07:21:46.196048477 +0000 UTC m=+0.407172375 container start 9610329a93167ffd1303e701d7536eb80c10313867fc6763bbec90b14ddef0c5 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1-deactivate, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-17T07:21:46.544 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:46 smithi044 podman[67578]: 2024-09-17 07:21:46.317505468 +0000 UTC m=+0.528629364 container attach 9610329a93167ffd1303e701d7536eb80c10313867fc6763bbec90b14ddef0c5 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-17T07:21:46.544 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:46 smithi044 podman[67578]: 2024-09-17 07:21:46.385964528 +0000 UTC m=+0.597088420 container died 9610329a93167ffd1303e701d7536eb80c10313867fc6763bbec90b14ddef0c5 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-17T07:21:46.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:46 smithi044 ceph-mon[58191]: pgmap v71: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:21:46.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:46 smithi044 ceph-mon[58191]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T07:21:46.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:46 smithi044 ceph-mon[58191]: osdmap e82: 8 total, 7 up, 8 in 2024-09-17T07:21:46.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:46 smithi044 ceph-mon[60356]: pgmap v71: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:21:46.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:46 smithi044 ceph-mon[60356]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T07:21:46.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:46 smithi044 ceph-mon[60356]: osdmap e82: 8 total, 7 up, 8 in 2024-09-17T07:21:46.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:46 smithi179 ceph-mon[50085]: pgmap v71: 161 pgs: 161 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:21:46.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:46 smithi179 ceph-mon[50085]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T07:21:46.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:46 smithi179 ceph-mon[50085]: osdmap e82: 8 total, 7 up, 8 in 2024-09-17T07:21:47.277 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:47 smithi044 podman[67601]: 2024-09-17 07:21:47.021533939 +0000 UTC m=+0.629463754 container remove 9610329a93167ffd1303e701d7536eb80c10313867fc6763bbec90b14ddef0c5 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=reef-19e2a2b, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-17T07:21:47.278 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:47 smithi044 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.1.service: Deactivated successfully. 2024-09-17T07:21:47.278 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:47 smithi044 systemd[1]: Stopped Ceph osd.1 for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:21:47.278 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:47 smithi044 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.1.service: Consumed 1.154s CPU time. 2024-09-17T07:21:47.539 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:47 smithi044 systemd[1]: Starting Ceph osd.1 for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:21:47.539 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:47 smithi044 podman[67703]: 2024-09-17 07:21:47.395085583 +0000 UTC m=+0.016738652 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:21:47.539 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:47 smithi044 podman[67703]: 2024-09-17 07:21:47.539535091 +0000 UTC m=+0.161188145 container create 0f0ba844776c4bc41f243c4cbc8d62af4f8f47a8d10789246a7f8fe20161edf7 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1-activate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-17T07:21:47.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:47 smithi044 ceph-mon[58191]: osdmap e83: 8 total, 7 up, 8 in 2024-09-17T07:21:47.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:47 smithi044 ceph-mon[60356]: osdmap e83: 8 total, 7 up, 8 in 2024-09-17T07:21:47.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:47 smithi179 ceph-mon[50085]: osdmap e83: 8 total, 7 up, 8 in 2024-09-17T07:21:48.268 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:47 smithi044 podman[67703]: 2024-09-17 07:21:47.959821244 +0000 UTC m=+0.581474317 container init 0f0ba844776c4bc41f243c4cbc8d62af4f8f47a8d10789246a7f8fe20161edf7 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1-activate, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-17T07:21:48.268 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:47 smithi044 podman[67703]: 2024-09-17 07:21:47.963522395 +0000 UTC m=+0.585175449 container start 0f0ba844776c4bc41f243c4cbc8d62af4f8f47a8d10789246a7f8fe20161edf7 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1-activate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-17T07:21:48.269 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:48 smithi044 podman[67703]: 2024-09-17 07:21:48.042734475 +0000 UTC m=+0.664387544 container attach 0f0ba844776c4bc41f243c4cbc8d62af4f8f47a8d10789246a7f8fe20161edf7 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1-activate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T07:21:48.554 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1-activate[67718]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-17T07:21:48.554 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:48 smithi044 bash[67703]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-17T07:21:48.554 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1-activate[67718]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-17T07:21:48.554 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:48 smithi044 bash[67703]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-17T07:21:48.554 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1-activate[67718]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-17T07:21:48.555 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:48 smithi044 bash[67703]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-17T07:21:48.555 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1-activate[67718]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-17T07:21:48.555 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:48 smithi044 bash[67703]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-17T07:21:48.555 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1-activate[67718]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-17T07:21:48.555 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:48 smithi044 bash[67703]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-17T07:21:48.555 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1-activate[67718]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-17T07:21:48.555 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:48 smithi044 bash[67703]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-17T07:21:48.555 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1-activate[67718]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-17T07:21:48.555 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:48 smithi044 bash[67703]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-17T07:21:48.555 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:48 smithi044 podman[67703]: 2024-09-17 07:21:48.319212628 +0000 UTC m=+0.940865682 container died 0f0ba844776c4bc41f243c4cbc8d62af4f8f47a8d10789246a7f8fe20161edf7 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1-activate, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-17T07:21:48.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:48 smithi044 ceph-mon[58191]: pgmap v74: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:21:48.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:48 smithi044 ceph-mon[58191]: Health check failed: Reduced data availability: 9 pgs inactive, 15 pgs peering (PG_AVAILABILITY) 2024-09-17T07:21:48.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:48 smithi044 ceph-mon[60356]: pgmap v74: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:21:48.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:48 smithi044 ceph-mon[60356]: Health check failed: Reduced data availability: 9 pgs inactive, 15 pgs peering (PG_AVAILABILITY) 2024-09-17T07:21:48.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:48 smithi179 ceph-mon[50085]: pgmap v74: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:21:48.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:48 smithi179 ceph-mon[50085]: Health check failed: Reduced data availability: 9 pgs inactive, 15 pgs peering (PG_AVAILABILITY) 2024-09-17T07:21:49.274 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:48 smithi044 podman[67832]: 2024-09-17 07:21:48.938124093 +0000 UTC m=+0.610155829 container remove 0f0ba844776c4bc41f243c4cbc8d62af4f8f47a8d10789246a7f8fe20161edf7 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1-activate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-17T07:21:49.275 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:49 smithi044 podman[67889]: 2024-09-17 07:21:49.055577444 +0000 UTC m=+0.017096907 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:21:49.275 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:49 smithi044 podman[67889]: 2024-09-17 07:21:49.1640898 +0000 UTC m=+0.125609251 container create 7655e43d99443ab88f001c32cd5778554910f0bc7e2219421e4ac99ef5a5de5a (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T07:21:49.825 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:49 smithi044 podman[67889]: 2024-09-17 07:21:49.375047917 +0000 UTC m=+0.336567365 container init 7655e43d99443ab88f001c32cd5778554910f0bc7e2219421e4ac99ef5a5de5a (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T07:21:49.825 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:49 smithi044 podman[67889]: 2024-09-17 07:21:49.378317106 +0000 UTC m=+0.339836554 container start 7655e43d99443ab88f001c32cd5778554910f0bc7e2219421e4ac99ef5a5de5a (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T07:21:49.825 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:49 smithi044 ceph-osd[67908]: -- 172.21.15.44:0/315482646 <== mon.1 v2:172.21.15.44:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f2afaa11e0 con 0x55f2b0822000 2024-09-17T07:21:49.825 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:49 smithi044 bash[67889]: 7655e43d99443ab88f001c32cd5778554910f0bc7e2219421e4ac99ef5a5de5a 2024-09-17T07:21:49.826 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:49 smithi044 systemd[1]: Started Ceph osd.1 for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:21:50.216 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:49 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1[67904]: 2024-09-17T07:21:49.918+0000 7f0ae4013640 -1 Falling back to public interface 2024-09-17T07:21:50.790 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:50 smithi044 ceph-mon[60356]: pgmap v75: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:21:50.790 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:50 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:50.790 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:50 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:50.790 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:50 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:21:50.791 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:50 smithi044 ceph-mon[58191]: pgmap v75: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:21:50.791 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:50 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:50.791 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:50 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:50.791 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:50 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:21:50.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:50 smithi179 ceph-mon[50085]: pgmap v75: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:21:50.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:50 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:50.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:50 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:50.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:50 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:21:51.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:51 smithi044 ceph-mon[58191]: Health check failed: Degraded data redundancy: 129/999 objects degraded (12.913%), 22 pgs degraded (PG_DEGRADED) 2024-09-17T07:21:51.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:51 smithi044 ceph-mon[58191]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 15 pgs peering) 2024-09-17T07:21:51.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:51 smithi044 ceph-mon[60356]: Health check failed: Degraded data redundancy: 129/999 objects degraded (12.913%), 22 pgs degraded (PG_DEGRADED) 2024-09-17T07:21:51.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:51 smithi044 ceph-mon[60356]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 15 pgs peering) 2024-09-17T07:21:51.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:51 smithi179 ceph-mon[50085]: Health check failed: Degraded data redundancy: 129/999 objects degraded (12.913%), 22 pgs degraded (PG_DEGRADED) 2024-09-17T07:21:51.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:51 smithi179 ceph-mon[50085]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 15 pgs peering) 2024-09-17T07:21:52.825 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1[67904]: 2024-09-17T07:21:52.665+0000 7f0ae4013640 -1 osd.1 0 read_superblock omap replica is missing. 2024-09-17T07:21:52.825 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:52 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1[67904]: 2024-09-17T07:21:52.698+0000 7f0ae4013640 -1 osd.1 81 log_to_monitors true 2024-09-17T07:21:52.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:52 smithi044 ceph-mon[58191]: pgmap v76: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 604 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 129/999 objects degraded (12.913%) 2024-09-17T07:21:52.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:52 smithi044 ceph-mon[58191]: from='client.44517 172.21.15.44:0/3632851548' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:21:52.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:52 smithi044 ceph-mon[58191]: from='client.44523 172.21.15.44:0/4148597090' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/1355368272"}]: access denied 2024-09-17T07:21:52.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:52 smithi044 ceph-mon[60356]: pgmap v76: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 604 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 129/999 objects degraded (12.913%) 2024-09-17T07:21:52.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:52 smithi044 ceph-mon[60356]: from='client.44517 172.21.15.44:0/3632851548' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:21:52.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:52 smithi044 ceph-mon[60356]: from='client.44523 172.21.15.44:0/4148597090' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/1355368272"}]: access denied 2024-09-17T07:21:52.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:52 smithi179 ceph-mon[50085]: pgmap v76: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 604 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 129/999 objects degraded (12.913%) 2024-09-17T07:21:52.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:52 smithi179 ceph-mon[50085]: from='client.44517 172.21.15.44:0/3632851548' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:21:52.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:52 smithi179 ceph-mon[50085]: from='client.44523 172.21.15.44:0/4148597090' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/1355368272"}]: access denied 2024-09-17T07:21:53.823 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:53 smithi044 ceph-mon[58191]: from='osd.1 [v2:172.21.15.44:6810/1792992627,v1:172.21.15.44:6811/1792992627]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T07:21:53.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:53 smithi044 ceph-mon[60356]: from='osd.1 [v2:172.21.15.44:6810/1792992627,v1:172.21.15.44:6811/1792992627]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T07:21:53.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:53 smithi179 ceph-mon[50085]: from='osd.1 [v2:172.21.15.44:6810/1792992627,v1:172.21.15.44:6811/1792992627]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T07:21:54.074 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:21:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1[67904]: 2024-09-17T07:21:54.047+0000 7f0ad7db5640 -1 osd.1 81 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T07:21:54.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:54 smithi044 ceph-mon[60356]: pgmap v77: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 604 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 129/999 objects degraded (12.913%) 2024-09-17T07:21:54.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:54 smithi044 ceph-mon[60356]: from='osd.1 [v2:172.21.15.44:6810/1792992627,v1:172.21.15.44:6811/1792992627]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-17T07:21:54.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:54 smithi044 ceph-mon[60356]: osdmap e84: 8 total, 7 up, 8 in 2024-09-17T07:21:54.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:54 smithi044 ceph-mon[60356]: from='osd.1 [v2:172.21.15.44:6810/1792992627,v1:172.21.15.44:6811/1792992627]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-09-17T07:21:54.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:54 smithi044 ceph-mon[60356]: from='osd.1 [v2:172.21.15.44:6810/1792992627,v1:172.21.15.44:6811/1792992627]' entity='osd.1' 2024-09-17T07:21:54.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:54 smithi044 ceph-mon[58191]: pgmap v77: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 604 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 129/999 objects degraded (12.913%) 2024-09-17T07:21:54.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:54 smithi044 ceph-mon[58191]: from='osd.1 [v2:172.21.15.44:6810/1792992627,v1:172.21.15.44:6811/1792992627]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-17T07:21:54.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:54 smithi044 ceph-mon[58191]: osdmap e84: 8 total, 7 up, 8 in 2024-09-17T07:21:54.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:54 smithi044 ceph-mon[58191]: from='osd.1 [v2:172.21.15.44:6810/1792992627,v1:172.21.15.44:6811/1792992627]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-09-17T07:21:54.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:54 smithi044 ceph-mon[58191]: from='osd.1 [v2:172.21.15.44:6810/1792992627,v1:172.21.15.44:6811/1792992627]' entity='osd.1' 2024-09-17T07:21:54.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:54 smithi179 ceph-mon[50085]: pgmap v77: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 604 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 129/999 objects degraded (12.913%) 2024-09-17T07:21:54.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:54 smithi179 ceph-mon[50085]: from='osd.1 [v2:172.21.15.44:6810/1792992627,v1:172.21.15.44:6811/1792992627]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-17T07:21:54.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:54 smithi179 ceph-mon[50085]: osdmap e84: 8 total, 7 up, 8 in 2024-09-17T07:21:54.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:54 smithi179 ceph-mon[50085]: from='osd.1 [v2:172.21.15.44:6810/1792992627,v1:172.21.15.44:6811/1792992627]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-09-17T07:21:54.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:54 smithi179 ceph-mon[50085]: from='osd.1 [v2:172.21.15.44:6810/1792992627,v1:172.21.15.44:6811/1792992627]' entity='osd.1' 2024-09-17T07:21:55.804 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:55 smithi044 ceph-mon[60356]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T07:21:55.804 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:55 smithi044 ceph-mon[60356]: osd.1 [v2:172.21.15.44:6810/1792992627,v1:172.21.15.44:6811/1792992627] boot 2024-09-17T07:21:55.804 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:55 smithi044 ceph-mon[60356]: osdmap e85: 8 total, 8 up, 8 in 2024-09-17T07:21:55.805 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:55 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T07:21:55.805 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:55 smithi044 ceph-mon[58191]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T07:21:55.805 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:55 smithi044 ceph-mon[58191]: osd.1 [v2:172.21.15.44:6810/1792992627,v1:172.21.15.44:6811/1792992627] boot 2024-09-17T07:21:55.805 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:55 smithi044 ceph-mon[58191]: osdmap e85: 8 total, 8 up, 8 in 2024-09-17T07:21:55.805 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:55 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T07:21:55.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:55 smithi179 ceph-mon[50085]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T07:21:55.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:55 smithi179 ceph-mon[50085]: osd.1 [v2:172.21.15.44:6810/1792992627,v1:172.21.15.44:6811/1792992627] boot 2024-09-17T07:21:55.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:55 smithi179 ceph-mon[50085]: osdmap e85: 8 total, 8 up, 8 in 2024-09-17T07:21:55.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:55 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T07:21:56.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:56 smithi179 ceph-mon[50085]: pgmap v80: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 604 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 129/999 objects degraded (12.913%) 2024-09-17T07:21:56.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:56 smithi179 ceph-mon[50085]: osdmap e86: 8 total, 8 up, 8 in 2024-09-17T07:21:56.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:56 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:56.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:56 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:57.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:56 smithi044 ceph-mon[60356]: pgmap v80: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 604 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 129/999 objects degraded (12.913%) 2024-09-17T07:21:57.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:56 smithi044 ceph-mon[60356]: osdmap e86: 8 total, 8 up, 8 in 2024-09-17T07:21:57.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:56 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:57.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:56 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:57.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:56 smithi044 ceph-mon[58191]: pgmap v80: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 604 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 129/999 objects degraded (12.913%) 2024-09-17T07:21:57.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:56 smithi044 ceph-mon[58191]: osdmap e86: 8 total, 8 up, 8 in 2024-09-17T07:21:57.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:56 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:57.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:56 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:58.297 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:57 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:58.297 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:57 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:58.297 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:57 smithi044 ceph-mon[58191]: pgmap v82: 161 pgs: 38 peering, 18 active+undersized, 7 active+undersized+degraded, 98 active+clean; 604 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 52/999 objects degraded (5.205%) 2024-09-17T07:21:58.297 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:57 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:58.297 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:57 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:58.297 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:57 smithi044 ceph-mon[60356]: pgmap v82: 161 pgs: 38 peering, 18 active+undersized, 7 active+undersized+degraded, 98 active+clean; 604 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 52/999 objects degraded (5.205%) 2024-09-17T07:21:58.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:57 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:58.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:57 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:21:58.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:57 smithi179 ceph-mon[50085]: pgmap v82: 161 pgs: 38 peering, 18 active+undersized, 7 active+undersized+degraded, 98 active+clean; 604 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 52/999 objects degraded (5.205%) 2024-09-17T07:21:59.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:58 smithi044 ceph-mon[58191]: Health check update: Degraded data redundancy: 52/999 objects degraded (5.205%), 7 pgs degraded (PG_DEGRADED) 2024-09-17T07:21:59.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:58 smithi044 ceph-mon[60356]: Health check update: Degraded data redundancy: 52/999 objects degraded (5.205%), 7 pgs degraded (PG_DEGRADED) 2024-09-17T07:21:59.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:58 smithi179 ceph-mon[50085]: Health check update: Degraded data redundancy: 52/999 objects degraded (5.205%), 7 pgs degraded (PG_DEGRADED) 2024-09-17T07:21:59.954 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-09-17T07:22:00.199 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:21:59 smithi044 ceph-mon[58191]: pgmap v83: 161 pgs: 38 peering, 5 active+undersized, 1 active+undersized+degraded, 117 active+clean; 604 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 10/999 objects degraded (1.001%) 2024-09-17T07:22:00.200 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:21:59 smithi044 ceph-mon[60356]: pgmap v83: 161 pgs: 38 peering, 5 active+undersized, 1 active+undersized+degraded, 117 active+clean; 604 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 10/999 objects degraded (1.001%) 2024-09-17T07:22:00.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:21:59 smithi179 ceph-mon[50085]: pgmap v83: 161 pgs: 38 peering, 5 active+undersized, 1 active+undersized+degraded, 117 active+clean; 604 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 10/999 objects degraded (1.001%) 2024-09-17T07:22:00.642 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T07:22:00.642 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.a smithi044 *:9093,9094 running (3m) 4s ago 8m 18.9M - 0.20.0 0881eb8f169f 0473d4b023ad 2024-09-17T07:22:00.643 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.a smithi179 *:3000 running (3m) 88s ago 8m 27.2M - 6.7.4 557c83e11646 43ab0e03dcac 2024-09-17T07:22:00.643 INFO:teuthology.orchestra.run.smithi044.stdout:iscsi.foo.smithi044.elmtfk smithi044 unknown 4s ago 7m - - 2024-09-17T07:22:00.643 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.x smithi179 *:8443,8765 running (2m) 88s ago 12m 436M - 18.2.4-855-g19e2a2be 980a0bfcd684 66b52dc26022 2024-09-17T07:22:00.643 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.y smithi044 *:8443,8765 running (3m) 4s ago 13m 479M - 18.2.4-855-g19e2a2be 980a0bfcd684 8080dd318a1c 2024-09-17T07:22:00.643 INFO:teuthology.orchestra.run.smithi044.stdout:mon.a smithi044 running (2m) 4s ago 13m 45.2M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 c3293e8cdbba 2024-09-17T07:22:00.643 INFO:teuthology.orchestra.run.smithi044.stdout:mon.b smithi179 running (95s) 88s ago 12m 27.7M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 ade749759184 2024-09-17T07:22:00.643 INFO:teuthology.orchestra.run.smithi044.stdout:mon.c smithi044 running (114s) 4s ago 12m 38.7M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 02f85d31e0bd 2024-09-17T07:22:00.643 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.a smithi044 *:9100 running (3m) 4s ago 8m 17.1M - 0.18.1 e5a616e4b9cf 8a2cde5bd9ed 2024-09-17T07:22:00.643 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.b smithi179 *:9100 running (3m) 88s ago 9m 8929k - 0.18.1 e5a616e4b9cf 04c63b5cb524 2024-09-17T07:22:00.643 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (45s) 4s ago 12m 50.4M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 3ba47a4e2e95 2024-09-17T07:22:00.643 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (11s) 4s ago 11m 11.7M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 7655e43d9944 2024-09-17T07:22:00.643 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (11m) 4s ago 11m 44.7M 4096M 16.2.0 24ecd6d5f14c 7fd2f30be874 2024-09-17T07:22:00.644 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi044 running (11m) 4s ago 11m 50.1M 4096M 16.2.0 24ecd6d5f14c 2b7d623395a7 2024-09-17T07:22:00.644 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi179 running (10m) 88s ago 10m 51.7M 4096M 16.2.0 24ecd6d5f14c 911e6f47c822 2024-09-17T07:22:00.644 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi179 running (10m) 88s ago 10m 47.8M 4096M 16.2.0 24ecd6d5f14c e56060742dff 2024-09-17T07:22:00.644 INFO:teuthology.orchestra.run.smithi044.stdout:osd.6 smithi179 running (10m) 88s ago 10m 47.4M 4096M 16.2.0 24ecd6d5f14c 7a5c30b68cbc 2024-09-17T07:22:00.644 INFO:teuthology.orchestra.run.smithi044.stdout:osd.7 smithi179 running (9m) 88s ago 9m 53.5M 4096M 16.2.0 24ecd6d5f14c b0d186fbf8a6 2024-09-17T07:22:00.644 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.a smithi179 *:9095 unknown 88s ago 9m - - 2024-09-17T07:22:00.644 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi044.nhyhnh smithi044 *:8000 running (7m) 4s ago 7m 38.9M - 16.2.0 24ecd6d5f14c 9b455479614f 2024-09-17T07:22:00.644 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi179.tqlzyy smithi179 *:8000 running (7m) 88s ago 7m 41.2M - 16.2.0 24ecd6d5f14c c8d1693241ce 2024-09-17T07:22:00.644 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi044.vnufiz smithi044 *:80 running (7m) 4s ago 7m 38.9M - 16.2.0 24ecd6d5f14c 6eae2bff5ec8 2024-09-17T07:22:00.644 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi179.ecjhll smithi179 *:80 running (7m) 88s ago 7m 41.7M - 16.2.0 24ecd6d5f14c 253109ee0394 2024-09-17T07:22:00.824 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:22:00 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:22:00.691+0000 7f5a7ae47640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-09-17T07:22:01.045 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:22:01.046 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-09-17T07:22:01.046 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3 2024-09-17T07:22:01.046 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:22:01.046 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-09-17T07:22:01.046 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T07:22:01.046 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:22:01.046 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-09-17T07:22:01.046 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 6, 2024-09-17T07:22:01.046 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T07:22:01.046 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:22:01.046 INFO:teuthology.orchestra.run.smithi044.stdout: "rgw": { 2024-09-17T07:22:01.047 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2024-09-17T07:22:01.047 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:22:01.047 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-09-17T07:22:01.047 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 10, 2024-09-17T07:22:01.047 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 7 2024-09-17T07:22:01.047 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-09-17T07:22:01.047 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:22:01.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[58191]: from='client.44532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:22:01.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:01.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[58191]: from='client.44538 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:22:01.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:01.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:01.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:22:01.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:22:01.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[58191]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:22:01.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[58191]: pgmap v84: 161 pgs: 38 peering, 123 active+clean; 604 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 146 B/s rd, 0 op/s 2024-09-17T07:22:01.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[58191]: pgmap v85: 161 pgs: 38 peering, 123 active+clean; 604 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 185 B/s rd, 0 op/s 2024-09-17T07:22:01.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:01.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[58191]: from='client.54427 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:22:01.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:22:01.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:22:01.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:22:01.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:22:01.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-17T07:22:01.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[58191]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-17T07:22:01.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[58191]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-09-17T07:22:01.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[58191]: from='client.54433 172.21.15.44:0/2137080573' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:22:01.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[60356]: from='client.44532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:22:01.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:01.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[60356]: from='client.44538 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:22:01.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:01.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:01.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:22:01.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:22:01.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[60356]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:22:01.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[60356]: pgmap v84: 161 pgs: 38 peering, 123 active+clean; 604 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 146 B/s rd, 0 op/s 2024-09-17T07:22:01.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[60356]: pgmap v85: 161 pgs: 38 peering, 123 active+clean; 604 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 185 B/s rd, 0 op/s 2024-09-17T07:22:01.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:01.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[60356]: from='client.54427 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:22:01.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:22:01.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:22:01.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:22:01.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:22:01.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-17T07:22:01.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[60356]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-17T07:22:01.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[60356]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-09-17T07:22:01.329 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:01 smithi044 ceph-mon[60356]: from='client.54433 172.21.15.44:0/2137080573' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:22:01.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:01 smithi179 ceph-mon[50085]: from='client.44532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:22:01.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:01 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:01.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:01 smithi179 ceph-mon[50085]: from='client.44538 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:22:01.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:01 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:01.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:01 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:01.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:01 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:22:01.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:01 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:22:01.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:01 smithi179 ceph-mon[50085]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:22:01.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:01 smithi179 ceph-mon[50085]: pgmap v84: 161 pgs: 38 peering, 123 active+clean; 604 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 146 B/s rd, 0 op/s 2024-09-17T07:22:01.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:01 smithi179 ceph-mon[50085]: pgmap v85: 161 pgs: 38 peering, 123 active+clean; 604 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 185 B/s rd, 0 op/s 2024-09-17T07:22:01.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:01 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:01.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:01 smithi179 ceph-mon[50085]: from='client.54427 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:22:01.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:01 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:22:01.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:01 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:22:01.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:01 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:22:01.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:01 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:22:01.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:01 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-17T07:22:01.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:01 smithi179 ceph-mon[50085]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-17T07:22:01.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:01 smithi179 ceph-mon[50085]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-09-17T07:22:01.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:01 smithi179 ceph-mon[50085]: from='client.54433 172.21.15.44:0/2137080573' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:22:01.382 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:22:01.383 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9", 2024-09-17T07:22:01.383 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-09-17T07:22:01.383 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T07:22:01.383 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [ 2024-09-17T07:22:01.383 INFO:teuthology.orchestra.run.smithi044.stdout: "mon", 2024-09-17T07:22:01.383 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr" 2024-09-17T07:22:01.383 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-09-17T07:22:01.383 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "7/23 daemons upgraded", 2024-09-17T07:22:01.383 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "Currently upgrading osd daemons", 2024-09-17T07:22:01.384 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-09-17T07:22:01.384 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:22:01.784 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_WARN 2 failed cephadm daemon(s) 2024-09-17T07:22:01.784 INFO:teuthology.orchestra.run.smithi044.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T07:22:01.784 INFO:teuthology.orchestra.run.smithi044.stdout: daemon iscsi.foo.smithi044.elmtfk on smithi044 is in unknown state 2024-09-17T07:22:01.784 INFO:teuthology.orchestra.run.smithi044.stdout: daemon prometheus.a on smithi179 is in unknown state 2024-09-17T07:22:02.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:02 smithi044 ceph-mon[58191]: from='client.44556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:22:02.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:02 smithi044 ceph-mon[58191]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/999 objects degraded (1.001%), 1 pg degraded) 2024-09-17T07:22:02.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:02 smithi044 ceph-mon[58191]: from='client.54445 172.21.15.44:0/1216725290' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:22:02.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:02 smithi044 ceph-mon[60356]: from='client.44556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:22:02.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:02 smithi044 ceph-mon[60356]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/999 objects degraded (1.001%), 1 pg degraded) 2024-09-17T07:22:02.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:02 smithi044 ceph-mon[60356]: from='client.54445 172.21.15.44:0/1216725290' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:22:02.363 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:02 smithi179 ceph-mon[50085]: from='client.44556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:22:02.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:02 smithi179 ceph-mon[50085]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/999 objects degraded (1.001%), 1 pg degraded) 2024-09-17T07:22:02.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:02 smithi179 ceph-mon[50085]: from='client.54445 172.21.15.44:0/1216725290' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:22:03.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:03 smithi044 ceph-mon[58191]: pgmap v86: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 136 B/s rd, 0 op/s 2024-09-17T07:22:03.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:03 smithi044 ceph-mon[60356]: pgmap v86: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 136 B/s rd, 0 op/s 2024-09-17T07:22:03.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:03 smithi179 ceph-mon[50085]: pgmap v86: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 136 B/s rd, 0 op/s 2024-09-17T07:22:06.067 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:05 smithi179 ceph-mon[50085]: pgmap v87: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:22:06.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:05 smithi044 ceph-mon[58191]: pgmap v87: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:22:06.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:05 smithi044 ceph-mon[60356]: pgmap v87: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:22:06.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:06 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/1531898557' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:22:06.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:06 smithi044 ceph-mon[58191]: from='client.44586 172.21.15.44:0/1594027666' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:22:06.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:06 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/1531898557' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:22:06.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:06 smithi044 ceph-mon[60356]: from='client.44586 172.21.15.44:0/1594027666' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:22:07.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:06 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/1531898557' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:22:07.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:06 smithi179 ceph-mon[50085]: from='client.44586 172.21.15.44:0/1594027666' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:22:07.749 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:07 smithi044 ceph-mon[58191]: pgmap v88: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 107 B/s rd, 0 op/s 2024-09-17T07:22:07.750 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:07 smithi044 ceph-mon[60356]: pgmap v88: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 107 B/s rd, 0 op/s 2024-09-17T07:22:08.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:07 smithi179 ceph-mon[50085]: pgmap v88: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 107 B/s rd, 0 op/s 2024-09-17T07:22:10.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:09 smithi044 ceph-mon[58191]: pgmap v89: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 107 B/s rd, 0 op/s 2024-09-17T07:22:10.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:09 smithi044 ceph-mon[60356]: pgmap v89: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 107 B/s rd, 0 op/s 2024-09-17T07:22:10.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:09 smithi179 ceph-mon[50085]: pgmap v89: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 107 B/s rd, 0 op/s 2024-09-17T07:22:11.324 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:11 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:11.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:11 smithi044 ceph-mon[60356]: pgmap v90: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-09-17T07:22:11.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:11 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:11.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:11 smithi044 ceph-mon[58191]: pgmap v90: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-09-17T07:22:11.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:11 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:11.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:11 smithi179 ceph-mon[50085]: pgmap v90: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-09-17T07:22:14.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:13 smithi044 ceph-mon[60356]: pgmap v91: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:22:14.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:13 smithi044 ceph-mon[58191]: pgmap v91: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:22:14.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:13 smithi179 ceph-mon[50085]: pgmap v91: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:22:15.876 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:15 smithi044 ceph-mon[60356]: pgmap v92: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:22:15.876 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:15 smithi044 ceph-mon[58191]: pgmap v92: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:22:16.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:15 smithi179 ceph-mon[50085]: pgmap v92: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:22:16.707 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:16 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-17T07:22:16.707 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:16 smithi044 ceph-mon[58191]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-17T07:22:16.707 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:16 smithi044 ceph-mon[58191]: Upgrade: osd.2 is safe to restart 2024-09-17T07:22:16.707 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:16 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-17T07:22:16.707 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:16 smithi044 ceph-mon[60356]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-17T07:22:16.707 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:16 smithi044 ceph-mon[60356]: Upgrade: osd.2 is safe to restart 2024-09-17T07:22:17.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:16 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-17T07:22:17.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:16 smithi179 ceph-mon[50085]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-17T07:22:17.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:16 smithi179 ceph-mon[50085]: Upgrade: osd.2 is safe to restart 2024-09-17T07:22:17.634 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:17 smithi044 ceph-mon[58191]: pgmap v93: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:22:17.634 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:17 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:17.634 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:17 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-17T07:22:17.634 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:17 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:22:17.635 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:17 smithi044 ceph-mon[60356]: pgmap v93: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:22:17.635 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:17 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:17.635 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:17 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-17T07:22:17.635 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:17 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:22:18.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:17 smithi179 ceph-mon[50085]: pgmap v93: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:22:18.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:17 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:18.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:17 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-17T07:22:18.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:17 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:22:18.654 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:18 smithi044 ceph-mon[60356]: Upgrade: Updating osd.2 2024-09-17T07:22:18.654 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:18 smithi044 ceph-mon[60356]: Deploying daemon osd.2 on smithi044 2024-09-17T07:22:18.654 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:18 smithi044 ceph-mon[58191]: Upgrade: Updating osd.2 2024-09-17T07:22:18.654 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:18 smithi044 ceph-mon[58191]: Deploying daemon osd.2 on smithi044 2024-09-17T07:22:19.113 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:18 smithi179 ceph-mon[50085]: Upgrade: Updating osd.2 2024-09-17T07:22:19.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:18 smithi179 ceph-mon[50085]: Deploying daemon osd.2 on smithi044 2024-09-17T07:22:20.052 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:19 smithi044 ceph-mon[58191]: pgmap v94: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:22:20.052 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:19 smithi044 ceph-mon[60356]: pgmap v94: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:22:20.113 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:19 smithi179 ceph-mon[50085]: pgmap v94: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:22:20.574 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:20 smithi044 systemd[1]: Stopping Ceph osd.2 for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:22:21.317 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.2[34077]: debug 2024-09-17T07:22:20.927+0000 7f8aa8992700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-17T07:22:21.318 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.2[34077]: debug 2024-09-17T07:22:20.927+0000 7f8aa8992700 -1 osd.2 86 *** Got signal Terminated *** 2024-09-17T07:22:21.318 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.2[34077]: debug 2024-09-17T07:22:20.927+0000 7f8aa8992700 -1 osd.2 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T07:22:21.318 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:20 smithi044 podman[70487]: 2024-09-17 07:22:20.95927152 +0000 UTC m=+0.164189067 container died 7fd2f30be874eba71077e2896f5c38a8f5abc8f95f778bd3c418df5496850778 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.2, org.label-schema.name=CentOS Base Image, CEPH_POINT_RELEASE=-16.2.0, GIT_CLEAN=True, ceph=True, maintainer=Dimitri Savineau , org.label-schema.schema-version=1.0, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20201204) 2024-09-17T07:22:21.574 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:21 smithi044 podman[70487]: 2024-09-17 07:22:21.317390401 +0000 UTC m=+0.522307743 container cleanup 7fd2f30be874eba71077e2896f5c38a8f5abc8f95f778bd3c418df5496850778 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.2, org.label-schema.schema-version=1.0, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, RELEASE=HEAD, maintainer=Dimitri Savineau , GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-16.2.0, GIT_CLEAN=True) 2024-09-17T07:22:21.574 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:21 smithi044 bash[70487]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.2 2024-09-17T07:22:22.006 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:21 smithi044 podman[70534]: 2024-09-17 07:22:21.635753819 +0000 UTC m=+0.671676864 container remove 7fd2f30be874eba71077e2896f5c38a8f5abc8f95f778bd3c418df5496850778 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.2, maintainer=Dimitri Savineau , org.label-schema.build-date=20201204, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-16.2.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=HEAD, ceph=True, GIT_CLEAN=True, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557) 2024-09-17T07:22:22.007 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[58191]: pgmap v95: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:22:22.007 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[58191]: osd.2 reported immediately failed by osd.4 2024-09-17T07:22:22.007 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[58191]: osd.2 failed (root=default,host=smithi044) (connection refused reported by osd.4) 2024-09-17T07:22:22.007 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[58191]: osd.2 reported immediately failed by osd.7 2024-09-17T07:22:22.007 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[58191]: osd.2 reported immediately failed by osd.6 2024-09-17T07:22:22.007 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[58191]: osd.2 reported immediately failed by osd.0 2024-09-17T07:22:22.007 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[58191]: osd.2 reported immediately failed by osd.1 2024-09-17T07:22:22.007 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[58191]: osd.2 reported immediately failed by osd.1 2024-09-17T07:22:22.007 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[58191]: osd.2 reported immediately failed by osd.6 2024-09-17T07:22:22.007 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[58191]: osd.2 reported immediately failed by osd.5 2024-09-17T07:22:22.007 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[58191]: osd.2 reported immediately failed by osd.4 2024-09-17T07:22:22.007 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[58191]: osd.2 reported immediately failed by osd.0 2024-09-17T07:22:22.008 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[58191]: osd.2 reported immediately failed by osd.7 2024-09-17T07:22:22.008 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[58191]: osd.2 reported immediately failed by osd.5 2024-09-17T07:22:22.008 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[58191]: osd.2 reported immediately failed by osd.3 2024-09-17T07:22:22.008 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[58191]: osd.2 reported immediately failed by osd.3 2024-09-17T07:22:22.008 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[58191]: from='client.54487 172.21.15.44:0/1052928497' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:22:22.008 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[58191]: osd.2 reported immediately failed by osd.6 2024-09-17T07:22:22.008 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[58191]: osd.2 reported immediately failed by osd.0 2024-09-17T07:22:22.008 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[58191]: osd.2 reported immediately failed by osd.1 2024-09-17T07:22:22.008 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[58191]: osd.2 reported immediately failed by osd.4 2024-09-17T07:22:22.008 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[58191]: osd.2 reported immediately failed by osd.7 2024-09-17T07:22:22.008 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[58191]: osd.2 reported immediately failed by osd.1 2024-09-17T07:22:22.008 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[58191]: osd.2 reported immediately failed by osd.0 2024-09-17T07:22:22.008 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[58191]: osd.2 reported immediately failed by osd.6 2024-09-17T07:22:22.008 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[58191]: osd.2 reported immediately failed by osd.5 2024-09-17T07:22:22.009 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[58191]: osd.2 reported immediately failed by osd.4 2024-09-17T07:22:22.009 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[58191]: osd.2 reported immediately failed by osd.7 2024-09-17T07:22:22.009 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[58191]: osd.2 reported immediately failed by osd.5 2024-09-17T07:22:22.009 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[58191]: osd.2 reported immediately failed by osd.3 2024-09-17T07:22:22.009 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[58191]: osd.2 reported immediately failed by osd.3 2024-09-17T07:22:22.009 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[58191]: from='client.54493 172.21.15.44:0/1774760723' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:22:22.009 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[60356]: pgmap v95: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:22:22.009 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[60356]: osd.2 reported immediately failed by osd.4 2024-09-17T07:22:22.009 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[60356]: osd.2 failed (root=default,host=smithi044) (connection refused reported by osd.4) 2024-09-17T07:22:22.010 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[60356]: osd.2 reported immediately failed by osd.7 2024-09-17T07:22:22.010 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[60356]: osd.2 reported immediately failed by osd.6 2024-09-17T07:22:22.010 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[60356]: osd.2 reported immediately failed by osd.0 2024-09-17T07:22:22.010 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[60356]: osd.2 reported immediately failed by osd.1 2024-09-17T07:22:22.010 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[60356]: osd.2 reported immediately failed by osd.1 2024-09-17T07:22:22.010 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[60356]: osd.2 reported immediately failed by osd.6 2024-09-17T07:22:22.010 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[60356]: osd.2 reported immediately failed by osd.5 2024-09-17T07:22:22.010 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[60356]: osd.2 reported immediately failed by osd.4 2024-09-17T07:22:22.010 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[60356]: osd.2 reported immediately failed by osd.0 2024-09-17T07:22:22.010 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[60356]: osd.2 reported immediately failed by osd.7 2024-09-17T07:22:22.010 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[60356]: osd.2 reported immediately failed by osd.5 2024-09-17T07:22:22.010 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[60356]: osd.2 reported immediately failed by osd.3 2024-09-17T07:22:22.010 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[60356]: osd.2 reported immediately failed by osd.3 2024-09-17T07:22:22.010 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[60356]: from='client.54487 172.21.15.44:0/1052928497' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:22:22.010 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[60356]: osd.2 reported immediately failed by osd.6 2024-09-17T07:22:22.011 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[60356]: osd.2 reported immediately failed by osd.0 2024-09-17T07:22:22.011 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[60356]: osd.2 reported immediately failed by osd.1 2024-09-17T07:22:22.011 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[60356]: osd.2 reported immediately failed by osd.4 2024-09-17T07:22:22.011 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[60356]: osd.2 reported immediately failed by osd.7 2024-09-17T07:22:22.011 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[60356]: osd.2 reported immediately failed by osd.1 2024-09-17T07:22:22.011 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[60356]: osd.2 reported immediately failed by osd.0 2024-09-17T07:22:22.011 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[60356]: osd.2 reported immediately failed by osd.6 2024-09-17T07:22:22.011 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[60356]: osd.2 reported immediately failed by osd.5 2024-09-17T07:22:22.011 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[60356]: osd.2 reported immediately failed by osd.4 2024-09-17T07:22:22.011 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[60356]: osd.2 reported immediately failed by osd.7 2024-09-17T07:22:22.011 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[60356]: osd.2 reported immediately failed by osd.5 2024-09-17T07:22:22.011 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[60356]: osd.2 reported immediately failed by osd.3 2024-09-17T07:22:22.011 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[60356]: osd.2 reported immediately failed by osd.3 2024-09-17T07:22:22.011 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:21 smithi044 ceph-mon[60356]: from='client.54493 172.21.15.44:0/1774760723' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:22:22.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:21 smithi179 ceph-mon[50085]: pgmap v95: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:22:22.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:21 smithi179 ceph-mon[50085]: osd.2 reported immediately failed by osd.4 2024-09-17T07:22:22.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:21 smithi179 ceph-mon[50085]: osd.2 failed (root=default,host=smithi044) (connection refused reported by osd.4) 2024-09-17T07:22:22.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:21 smithi179 ceph-mon[50085]: osd.2 reported immediately failed by osd.7 2024-09-17T07:22:22.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:21 smithi179 ceph-mon[50085]: osd.2 reported immediately failed by osd.6 2024-09-17T07:22:22.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:21 smithi179 ceph-mon[50085]: osd.2 reported immediately failed by osd.0 2024-09-17T07:22:22.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:21 smithi179 ceph-mon[50085]: osd.2 reported immediately failed by osd.1 2024-09-17T07:22:22.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:21 smithi179 ceph-mon[50085]: osd.2 reported immediately failed by osd.1 2024-09-17T07:22:22.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:21 smithi179 ceph-mon[50085]: osd.2 reported immediately failed by osd.6 2024-09-17T07:22:22.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:21 smithi179 ceph-mon[50085]: osd.2 reported immediately failed by osd.5 2024-09-17T07:22:22.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:21 smithi179 ceph-mon[50085]: osd.2 reported immediately failed by osd.4 2024-09-17T07:22:22.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:21 smithi179 ceph-mon[50085]: osd.2 reported immediately failed by osd.0 2024-09-17T07:22:22.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:21 smithi179 ceph-mon[50085]: osd.2 reported immediately failed by osd.7 2024-09-17T07:22:22.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:21 smithi179 ceph-mon[50085]: osd.2 reported immediately failed by osd.5 2024-09-17T07:22:22.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:21 smithi179 ceph-mon[50085]: osd.2 reported immediately failed by osd.3 2024-09-17T07:22:22.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:21 smithi179 ceph-mon[50085]: osd.2 reported immediately failed by osd.3 2024-09-17T07:22:22.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:21 smithi179 ceph-mon[50085]: from='client.54487 172.21.15.44:0/1052928497' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:22:22.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:21 smithi179 ceph-mon[50085]: osd.2 reported immediately failed by osd.6 2024-09-17T07:22:22.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:21 smithi179 ceph-mon[50085]: osd.2 reported immediately failed by osd.0 2024-09-17T07:22:22.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:21 smithi179 ceph-mon[50085]: osd.2 reported immediately failed by osd.1 2024-09-17T07:22:22.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:21 smithi179 ceph-mon[50085]: osd.2 reported immediately failed by osd.4 2024-09-17T07:22:22.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:21 smithi179 ceph-mon[50085]: osd.2 reported immediately failed by osd.7 2024-09-17T07:22:22.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:21 smithi179 ceph-mon[50085]: osd.2 reported immediately failed by osd.1 2024-09-17T07:22:22.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:21 smithi179 ceph-mon[50085]: osd.2 reported immediately failed by osd.0 2024-09-17T07:22:22.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:21 smithi179 ceph-mon[50085]: osd.2 reported immediately failed by osd.6 2024-09-17T07:22:22.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:21 smithi179 ceph-mon[50085]: osd.2 reported immediately failed by osd.5 2024-09-17T07:22:22.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:21 smithi179 ceph-mon[50085]: osd.2 reported immediately failed by osd.4 2024-09-17T07:22:22.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:21 smithi179 ceph-mon[50085]: osd.2 reported immediately failed by osd.7 2024-09-17T07:22:22.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:21 smithi179 ceph-mon[50085]: osd.2 reported immediately failed by osd.5 2024-09-17T07:22:22.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:21 smithi179 ceph-mon[50085]: osd.2 reported immediately failed by osd.3 2024-09-17T07:22:22.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:21 smithi179 ceph-mon[50085]: osd.2 reported immediately failed by osd.3 2024-09-17T07:22:22.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:21 smithi179 ceph-mon[50085]: from='client.54493 172.21.15.44:0/1774760723' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:22:22.937 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:22 smithi044 ceph-mon[58191]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T07:22:22.937 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:22 smithi044 ceph-mon[58191]: osdmap e87: 8 total, 7 up, 8 in 2024-09-17T07:22:22.937 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:22 smithi044 ceph-mon[60356]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T07:22:22.937 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:22 smithi044 ceph-mon[60356]: osdmap e87: 8 total, 7 up, 8 in 2024-09-17T07:22:23.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:22 smithi179 ceph-mon[50085]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T07:22:23.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:22 smithi179 ceph-mon[50085]: osdmap e87: 8 total, 7 up, 8 in 2024-09-17T07:22:23.652 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:23 smithi044 podman[70728]: 2024-09-17 07:22:23.28839258 +0000 UTC m=+0.016450786 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:22:23.653 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:23 smithi044 podman[70728]: 2024-09-17 07:22:23.388283106 +0000 UTC m=+0.116341342 container create 538e92b471796d7223bd54ca85fd307a91c917e43025434a782df04e2566b39e (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-17T07:22:23.991 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:23 smithi044 ceph-mon[58191]: pgmap v97: 161 pgs: 14 stale+active+clean, 147 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-09-17T07:22:23.992 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:23 smithi044 ceph-mon[58191]: osdmap e88: 8 total, 7 up, 8 in 2024-09-17T07:22:23.992 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:23 smithi044 ceph-mon[60356]: pgmap v97: 161 pgs: 14 stale+active+clean, 147 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-09-17T07:22:23.992 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:23 smithi044 ceph-mon[60356]: osdmap e88: 8 total, 7 up, 8 in 2024-09-17T07:22:23.992 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:23 smithi044 podman[70728]: 2024-09-17 07:22:23.799608179 +0000 UTC m=+0.527666381 container init 538e92b471796d7223bd54ca85fd307a91c917e43025434a782df04e2566b39e (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2-deactivate, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True) 2024-09-17T07:22:23.992 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:23 smithi044 podman[70728]: 2024-09-17 07:22:23.803028774 +0000 UTC m=+0.531086966 container start 538e92b471796d7223bd54ca85fd307a91c917e43025434a782df04e2566b39e (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2-deactivate, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-17T07:22:23.992 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:23 smithi044 podman[70728]: 2024-09-17 07:22:23.874539538 +0000 UTC m=+0.602597745 container attach 538e92b471796d7223bd54ca85fd307a91c917e43025434a782df04e2566b39e (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-17T07:22:24.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:23 smithi179 ceph-mon[50085]: pgmap v97: 161 pgs: 14 stale+active+clean, 147 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-09-17T07:22:24.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:23 smithi179 ceph-mon[50085]: osdmap e88: 8 total, 7 up, 8 in 2024-09-17T07:22:24.324 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:23 smithi044 podman[70728]: 2024-09-17 07:22:23.992252243 +0000 UTC m=+0.720310436 container died 538e92b471796d7223bd54ca85fd307a91c917e43025434a782df04e2566b39e (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2-deactivate, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-17T07:22:24.871 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:24 smithi044 ceph-mon[58191]: Health check failed: Degraded data redundancy: 28/999 objects degraded (2.803%), 3 pgs degraded (PG_DEGRADED) 2024-09-17T07:22:24.871 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:24 smithi044 podman[70751]: 2024-09-17 07:22:24.620118785 +0000 UTC m=+0.620944521 container remove 538e92b471796d7223bd54ca85fd307a91c917e43025434a782df04e2566b39e (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2-deactivate, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-17T07:22:24.871 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:24 smithi044 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.2.service: Deactivated successfully. 2024-09-17T07:22:24.871 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:24 smithi044 systemd[1]: Stopped Ceph osd.2 for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:22:24.871 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:24 smithi044 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.2.service: Consumed 1.172s CPU time. 2024-09-17T07:22:24.871 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:24 smithi044 ceph-mon[60356]: Health check failed: Degraded data redundancy: 28/999 objects degraded (2.803%), 3 pgs degraded (PG_DEGRADED) 2024-09-17T07:22:25.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:24 smithi179 ceph-mon[50085]: Health check failed: Degraded data redundancy: 28/999 objects degraded (2.803%), 3 pgs degraded (PG_DEGRADED) 2024-09-17T07:22:25.179 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:24 smithi044 systemd[1]: Starting Ceph osd.2 for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:22:25.179 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:25 smithi044 podman[70849]: 2024-09-17 07:22:24.987264115 +0000 UTC m=+0.016698373 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:22:25.180 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:25 smithi044 podman[70849]: 2024-09-17 07:22:25.179573614 +0000 UTC m=+0.209007871 container create da1bc1b0800fbd74b36447f14714f28ebc0d09aacffb4746997419ff816a6452 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2-activate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True) 2024-09-17T07:22:25.524 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:25 smithi044 podman[70849]: 2024-09-17 07:22:25.424176997 +0000 UTC m=+0.453611267 container init da1bc1b0800fbd74b36447f14714f28ebc0d09aacffb4746997419ff816a6452 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2-activate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-17T07:22:25.524 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:25 smithi044 podman[70849]: 2024-09-17 07:22:25.427651731 +0000 UTC m=+0.457085995 container start da1bc1b0800fbd74b36447f14714f28ebc0d09aacffb4746997419ff816a6452 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2-activate, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-17T07:22:25.524 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:25 smithi044 podman[70849]: 2024-09-17 07:22:25.524206739 +0000 UTC m=+0.553640997 container attach da1bc1b0800fbd74b36447f14714f28ebc0d09aacffb4746997419ff816a6452 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2-activate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T07:22:25.782 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:25 smithi044 ceph-mon[58191]: pgmap v99: 161 pgs: 5 active+undersized, 12 stale+active+clean, 3 active+undersized+degraded, 141 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 28/999 objects degraded (2.803%) 2024-09-17T07:22:25.783 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:25 smithi044 ceph-mon[60356]: pgmap v99: 161 pgs: 5 active+undersized, 12 stale+active+clean, 3 active+undersized+degraded, 141 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 28/999 objects degraded (2.803%) 2024-09-17T07:22:25.783 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2-activate[70865]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-17T07:22:25.783 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:25 smithi044 bash[70849]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-17T07:22:25.783 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2-activate[70865]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-17T07:22:25.783 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:25 smithi044 bash[70849]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-17T07:22:25.783 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2-activate[70865]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-17T07:22:25.783 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:25 smithi044 bash[70849]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-17T07:22:25.783 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2-activate[70865]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-17T07:22:25.784 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:25 smithi044 bash[70849]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-17T07:22:25.784 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2-activate[70865]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-17T07:22:25.784 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:25 smithi044 bash[70849]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-17T07:22:25.784 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2-activate[70865]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-17T07:22:25.784 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:25 smithi044 bash[70849]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-17T07:22:25.784 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:25 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2-activate[70865]: --> ceph-volume raw activate successful for osd ID: 2 2024-09-17T07:22:25.784 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:25 smithi044 bash[70849]: --> ceph-volume raw activate successful for osd ID: 2 2024-09-17T07:22:26.074 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:25 smithi044 podman[70849]: 2024-09-17 07:22:25.783045973 +0000 UTC m=+0.812480239 container died da1bc1b0800fbd74b36447f14714f28ebc0d09aacffb4746997419ff816a6452 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2-activate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-17T07:22:26.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:25 smithi179 ceph-mon[50085]: pgmap v99: 161 pgs: 5 active+undersized, 12 stale+active+clean, 3 active+undersized+degraded, 141 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 28/999 objects degraded (2.803%) 2024-09-17T07:22:26.704 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:26 smithi044 podman[70985]: 2024-09-17 07:22:26.453454226 +0000 UTC m=+0.663753156 container remove da1bc1b0800fbd74b36447f14714f28ebc0d09aacffb4746997419ff816a6452 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2-activate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-17T07:22:26.704 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:26 smithi044 podman[71045]: 2024-09-17 07:22:26.574870349 +0000 UTC m=+0.017126031 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:22:26.973 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:26 smithi044 podman[71045]: 2024-09-17 07:22:26.704441354 +0000 UTC m=+0.146697059 container create 43a48289eadad873c5d4b9da17d57c13225faa0b2e2f9a64fbbcbe7b851e8a83 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux ) 2024-09-17T07:22:27.324 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:26 smithi044 podman[71045]: 2024-09-17 07:22:26.973871138 +0000 UTC m=+0.416126819 container init 43a48289eadad873c5d4b9da17d57c13225faa0b2e2f9a64fbbcbe7b851e8a83 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-17T07:22:27.324 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:26 smithi044 podman[71045]: 2024-09-17 07:22:26.977330027 +0000 UTC m=+0.419585694 container start 43a48289eadad873c5d4b9da17d57c13225faa0b2e2f9a64fbbcbe7b851e8a83 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T07:22:27.325 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:26 smithi044 ceph-osd[71064]: -- 172.21.15.44:0/1061404887 <== mon.0 v2:172.21.15.44:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a48d7e31e0 con 0x55a48d781c00 2024-09-17T07:22:27.325 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:27 smithi044 bash[71045]: 43a48289eadad873c5d4b9da17d57c13225faa0b2e2f9a64fbbcbe7b851e8a83 2024-09-17T07:22:27.325 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:27 smithi044 systemd[1]: Started Ceph osd.2 for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:22:27.792 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2[71060]: 2024-09-17T07:22:27.513+0000 7f81bbe11640 -1 Falling back to public interface 2024-09-17T07:22:27.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:27 smithi044 ceph-mon[58191]: pgmap v100: 161 pgs: 35 active+undersized, 13 active+undersized+degraded, 113 active+clean; 604 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 88/999 objects degraded (8.809%) 2024-09-17T07:22:27.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:27.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:27.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:27 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:22:27.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:27 smithi044 ceph-mon[60356]: pgmap v100: 161 pgs: 35 active+undersized, 13 active+undersized+degraded, 113 active+clean; 604 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 88/999 objects degraded (8.809%) 2024-09-17T07:22:27.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:27.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:27.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:27 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:22:28.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:27 smithi179 ceph-mon[50085]: pgmap v100: 161 pgs: 35 active+undersized, 13 active+undersized+degraded, 113 active+clean; 604 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 88/999 objects degraded (8.809%) 2024-09-17T07:22:28.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:28.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:28.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:27 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:22:29.959 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:29 smithi044 ceph-mon[60356]: pgmap v101: 161 pgs: 35 active+undersized, 13 active+undersized+degraded, 113 active+clean; 604 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 88/999 objects degraded (8.809%) 2024-09-17T07:22:29.959 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:29 smithi044 ceph-mon[58191]: pgmap v101: 161 pgs: 35 active+undersized, 13 active+undersized+degraded, 113 active+clean; 604 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 88/999 objects degraded (8.809%) 2024-09-17T07:22:30.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:29 smithi179 ceph-mon[50085]: pgmap v101: 161 pgs: 35 active+undersized, 13 active+undersized+degraded, 113 active+clean; 604 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 88/999 objects degraded (8.809%) 2024-09-17T07:22:30.574 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2[71060]: 2024-09-17T07:22:30.255+0000 7f81bbe11640 -1 osd.2 0 read_superblock omap replica is missing. 2024-09-17T07:22:30.574 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:30 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2[71060]: 2024-09-17T07:22:30.278+0000 7f81bbe11640 -1 osd.2 86 log_to_monitors true 2024-09-17T07:22:30.838 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:30 smithi044 ceph-mon[58191]: from='osd.2 [v2:172.21.15.44:6818/1858459232,v1:172.21.15.44:6819/1858459232]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-17T07:22:30.838 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:30 smithi044 ceph-mon[60356]: from='osd.2 [v2:172.21.15.44:6818/1858459232,v1:172.21.15.44:6819/1858459232]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-17T07:22:31.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:30 smithi179 ceph-mon[50085]: from='osd.2 [v2:172.21.15.44:6818/1858459232,v1:172.21.15.44:6819/1858459232]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-17T07:22:31.574 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:22:31 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2[71060]: 2024-09-17T07:22:31.177+0000 7f81afbb3640 -1 osd.2 86 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T07:22:32.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:31 smithi044 ceph-mon[58191]: pgmap v102: 161 pgs: 35 active+undersized, 13 active+undersized+degraded, 113 active+clean; 604 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 88/999 objects degraded (8.809%) 2024-09-17T07:22:32.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:31 smithi044 ceph-mon[58191]: from='osd.2 [v2:172.21.15.44:6818/1858459232,v1:172.21.15.44:6819/1858459232]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-17T07:22:32.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:31 smithi044 ceph-mon[58191]: osdmap e89: 8 total, 7 up, 8 in 2024-09-17T07:22:32.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:31 smithi044 ceph-mon[58191]: from='osd.2 [v2:172.21.15.44:6818/1858459232,v1:172.21.15.44:6819/1858459232]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-09-17T07:22:32.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:31 smithi044 ceph-mon[58191]: from='osd.2 [v2:172.21.15.44:6818/1858459232,v1:172.21.15.44:6819/1858459232]' entity='osd.2' 2024-09-17T07:22:32.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:31 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:32.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:31 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:32.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:31 smithi044 ceph-mon[60356]: pgmap v102: 161 pgs: 35 active+undersized, 13 active+undersized+degraded, 113 active+clean; 604 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 88/999 objects degraded (8.809%) 2024-09-17T07:22:32.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:31 smithi044 ceph-mon[60356]: from='osd.2 [v2:172.21.15.44:6818/1858459232,v1:172.21.15.44:6819/1858459232]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-17T07:22:32.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:31 smithi044 ceph-mon[60356]: osdmap e89: 8 total, 7 up, 8 in 2024-09-17T07:22:32.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:31 smithi044 ceph-mon[60356]: from='osd.2 [v2:172.21.15.44:6818/1858459232,v1:172.21.15.44:6819/1858459232]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-09-17T07:22:32.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:31 smithi044 ceph-mon[60356]: from='osd.2 [v2:172.21.15.44:6818/1858459232,v1:172.21.15.44:6819/1858459232]' entity='osd.2' 2024-09-17T07:22:32.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:31 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:32.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:31 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:32.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:31 smithi179 ceph-mon[50085]: pgmap v102: 161 pgs: 35 active+undersized, 13 active+undersized+degraded, 113 active+clean; 604 KiB data, 144 MiB used, 715 GiB / 715 GiB avail; 88/999 objects degraded (8.809%) 2024-09-17T07:22:32.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:31 smithi179 ceph-mon[50085]: from='osd.2 [v2:172.21.15.44:6818/1858459232,v1:172.21.15.44:6819/1858459232]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-17T07:22:32.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:31 smithi179 ceph-mon[50085]: osdmap e89: 8 total, 7 up, 8 in 2024-09-17T07:22:32.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:31 smithi179 ceph-mon[50085]: from='osd.2 [v2:172.21.15.44:6818/1858459232,v1:172.21.15.44:6819/1858459232]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-09-17T07:22:32.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:31 smithi179 ceph-mon[50085]: from='osd.2 [v2:172.21.15.44:6818/1858459232,v1:172.21.15.44:6819/1858459232]' entity='osd.2' 2024-09-17T07:22:32.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:31 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:32.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:31 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:32.143 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-09-17T07:22:32.831 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:32 smithi044 ceph-mon[58191]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T07:22:32.831 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:32 smithi044 ceph-mon[58191]: osd.2 [v2:172.21.15.44:6818/1858459232,v1:172.21.15.44:6819/1858459232] boot 2024-09-17T07:22:32.831 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:32 smithi044 ceph-mon[58191]: osdmap e90: 8 total, 8 up, 8 in 2024-09-17T07:22:32.831 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:32 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:22:32.831 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:32 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:32.831 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:32 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:32.832 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:32 smithi044 ceph-mon[60356]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T07:22:32.832 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:32 smithi044 ceph-mon[60356]: osd.2 [v2:172.21.15.44:6818/1858459232,v1:172.21.15.44:6819/1858459232] boot 2024-09-17T07:22:32.832 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:32 smithi044 ceph-mon[60356]: osdmap e90: 8 total, 8 up, 8 in 2024-09-17T07:22:32.832 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:32 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:22:32.832 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:32 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:32.832 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:32 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:32.860 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T07:22:32.860 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.a smithi044 *:9093,9094 running (4m) 1s ago 8m 18.9M - 0.20.0 0881eb8f169f 0473d4b023ad 2024-09-17T07:22:32.860 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.a smithi179 *:3000 running (3m) 2m ago 8m 27.2M - 6.7.4 557c83e11646 43ab0e03dcac 2024-09-17T07:22:32.861 INFO:teuthology.orchestra.run.smithi044.stdout:iscsi.foo.smithi044.elmtfk smithi044 unknown 1s ago 8m - - 2024-09-17T07:22:32.861 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.x smithi179 *:8443,8765 running (3m) 2m ago 12m 436M - 18.2.4-855-g19e2a2be 980a0bfcd684 66b52dc26022 2024-09-17T07:22:32.861 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.y smithi044 *:8443,8765 running (3m) 1s ago 14m 481M - 18.2.4-855-g19e2a2be 980a0bfcd684 8080dd318a1c 2024-09-17T07:22:32.861 INFO:teuthology.orchestra.run.smithi044.stdout:mon.a smithi044 running (2m) 1s ago 14m 47.2M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 c3293e8cdbba 2024-09-17T07:22:32.861 INFO:teuthology.orchestra.run.smithi044.stdout:mon.b smithi179 running (2m) 2m ago 13m 27.7M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 ade749759184 2024-09-17T07:22:32.861 INFO:teuthology.orchestra.run.smithi044.stdout:mon.c smithi044 running (2m) 1s ago 13m 40.9M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 02f85d31e0bd 2024-09-17T07:22:32.861 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.a smithi044 *:9100 running (4m) 1s ago 9m 17.1M - 0.18.1 e5a616e4b9cf 8a2cde5bd9ed 2024-09-17T07:22:32.861 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.b smithi179 *:9100 running (4m) 2m ago 9m 8929k - 0.18.1 e5a616e4b9cf 04c63b5cb524 2024-09-17T07:22:32.861 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (77s) 1s ago 12m 51.9M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 3ba47a4e2e95 2024-09-17T07:22:32.861 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (43s) 1s ago 12m 49.2M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 7655e43d9944 2024-09-17T07:22:32.861 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (6s) 1s ago 12m 12.1M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 43a48289eada 2024-09-17T07:22:32.861 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi044 running (11m) 1s ago 11m 50.6M 4096M 16.2.0 24ecd6d5f14c 2b7d623395a7 2024-09-17T07:22:32.861 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi179 running (11m) 2m ago 11m 51.7M 4096M 16.2.0 24ecd6d5f14c 911e6f47c822 2024-09-17T07:22:32.862 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi179 running (11m) 2m ago 11m 47.8M 4096M 16.2.0 24ecd6d5f14c e56060742dff 2024-09-17T07:22:32.862 INFO:teuthology.orchestra.run.smithi044.stdout:osd.6 smithi179 running (10m) 2m ago 10m 47.4M 4096M 16.2.0 24ecd6d5f14c 7a5c30b68cbc 2024-09-17T07:22:32.862 INFO:teuthology.orchestra.run.smithi044.stdout:osd.7 smithi179 running (10m) 2m ago 10m 53.5M 4096M 16.2.0 24ecd6d5f14c b0d186fbf8a6 2024-09-17T07:22:32.862 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.a smithi179 *:9095 unknown 2m ago 9m - - 2024-09-17T07:22:32.862 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi044.nhyhnh smithi044 *:8000 running (8m) 1s ago 8m 39.0M - 16.2.0 24ecd6d5f14c 9b455479614f 2024-09-17T07:22:32.862 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi179.tqlzyy smithi179 *:8000 running (8m) 2m ago 8m 41.2M - 16.2.0 24ecd6d5f14c c8d1693241ce 2024-09-17T07:22:32.862 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi044.vnufiz smithi044 *:80 running (8m) 1s ago 8m 39.1M - 16.2.0 24ecd6d5f14c 6eae2bff5ec8 2024-09-17T07:22:32.862 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi179.ecjhll smithi179 *:80 running (8m) 2m ago 8m 41.7M - 16.2.0 24ecd6d5f14c 253109ee0394 2024-09-17T07:22:33.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:32 smithi179 ceph-mon[50085]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T07:22:33.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:32 smithi179 ceph-mon[50085]: osd.2 [v2:172.21.15.44:6818/1858459232,v1:172.21.15.44:6819/1858459232] boot 2024-09-17T07:22:33.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:32 smithi179 ceph-mon[50085]: osdmap e90: 8 total, 8 up, 8 in 2024-09-17T07:22:33.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:32 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T07:22:33.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:32 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:33.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:32 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:33.292 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:22:33.293 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-09-17T07:22:33.293 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3 2024-09-17T07:22:33.293 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:22:33.293 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-09-17T07:22:33.293 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T07:22:33.293 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:22:33.293 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-09-17T07:22:33.293 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 5, 2024-09-17T07:22:33.293 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3 2024-09-17T07:22:33.293 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:22:33.293 INFO:teuthology.orchestra.run.smithi044.stdout: "rgw": { 2024-09-17T07:22:33.293 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2024-09-17T07:22:33.294 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:22:33.294 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-09-17T07:22:33.294 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 9, 2024-09-17T07:22:33.294 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 8 2024-09-17T07:22:33.294 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-09-17T07:22:33.294 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:22:33.636 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:22:33.636 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9", 2024-09-17T07:22:33.637 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-09-17T07:22:33.637 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T07:22:33.637 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [ 2024-09-17T07:22:33.637 INFO:teuthology.orchestra.run.smithi044.stdout: "mon", 2024-09-17T07:22:33.637 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr" 2024-09-17T07:22:33.637 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-09-17T07:22:33.637 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "8/23 daemons upgraded", 2024-09-17T07:22:33.637 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "Currently upgrading osd daemons", 2024-09-17T07:22:33.637 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-09-17T07:22:33.637 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:22:33.884 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:33 smithi044 ceph-mon[58191]: from='client.54505 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:22:33.884 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:33 smithi044 ceph-mon[58191]: from='client.54511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:22:33.885 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:33 smithi044 ceph-mon[58191]: pgmap v105: 161 pgs: 35 active+undersized, 13 active+undersized+degraded, 113 active+clean; 604 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 88/999 objects degraded (8.809%) 2024-09-17T07:22:33.885 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:33 smithi044 ceph-mon[58191]: from='client.44634 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:22:33.885 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:33 smithi044 ceph-mon[58191]: osdmap e91: 8 total, 8 up, 8 in 2024-09-17T07:22:33.885 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:33 smithi044 ceph-mon[58191]: from='client.54523 172.21.15.44:0/2673971454' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:22:33.885 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:33 smithi044 ceph-mon[58191]: Health check update: Degraded data redundancy: 88/999 objects degraded (8.809%), 13 pgs degraded (PG_DEGRADED) 2024-09-17T07:22:33.885 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:33 smithi044 ceph-mon[60356]: from='client.54505 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:22:33.885 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:33 smithi044 ceph-mon[60356]: from='client.54511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:22:33.885 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:33 smithi044 ceph-mon[60356]: pgmap v105: 161 pgs: 35 active+undersized, 13 active+undersized+degraded, 113 active+clean; 604 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 88/999 objects degraded (8.809%) 2024-09-17T07:22:33.885 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:33 smithi044 ceph-mon[60356]: from='client.44634 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:22:33.885 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:33 smithi044 ceph-mon[60356]: osdmap e91: 8 total, 8 up, 8 in 2024-09-17T07:22:33.885 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:33 smithi044 ceph-mon[60356]: from='client.54523 172.21.15.44:0/2673971454' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:22:33.886 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:33 smithi044 ceph-mon[60356]: Health check update: Degraded data redundancy: 88/999 objects degraded (8.809%), 13 pgs degraded (PG_DEGRADED) 2024-09-17T07:22:34.066 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_WARN 2 failed cephadm daemon(s); Degraded data redundancy: 88/999 objects degraded (8.809%), 13 pgs degraded 2024-09-17T07:22:34.067 INFO:teuthology.orchestra.run.smithi044.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T07:22:34.067 INFO:teuthology.orchestra.run.smithi044.stdout: daemon iscsi.foo.smithi044.elmtfk on smithi044 is in unknown state 2024-09-17T07:22:34.067 INFO:teuthology.orchestra.run.smithi044.stdout: daemon prometheus.a on smithi179 is in unknown state 2024-09-17T07:22:34.067 INFO:teuthology.orchestra.run.smithi044.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 88/999 objects degraded (8.809%), 13 pgs degraded 2024-09-17T07:22:34.067 INFO:teuthology.orchestra.run.smithi044.stdout: pg 2.3 is active+undersized+degraded, acting [5,7] 2024-09-17T07:22:34.067 INFO:teuthology.orchestra.run.smithi044.stdout: pg 2.a is active+undersized+degraded, acting [1,3] 2024-09-17T07:22:34.067 INFO:teuthology.orchestra.run.smithi044.stdout: pg 2.17 is active+undersized+degraded, acting [6,5] 2024-09-17T07:22:34.067 INFO:teuthology.orchestra.run.smithi044.stdout: pg 3.0 is active+undersized+degraded, acting [1,6] 2024-09-17T07:22:34.067 INFO:teuthology.orchestra.run.smithi044.stdout: pg 3.4 is active+undersized+degraded, acting [1,5] 2024-09-17T07:22:34.067 INFO:teuthology.orchestra.run.smithi044.stdout: pg 3.5 is active+undersized+degraded, acting [5,3] 2024-09-17T07:22:34.067 INFO:teuthology.orchestra.run.smithi044.stdout: pg 3.9 is active+undersized+degraded, acting [4,7] 2024-09-17T07:22:34.067 INFO:teuthology.orchestra.run.smithi044.stdout: pg 3.13 is active+undersized+degraded, acting [7,4] 2024-09-17T07:22:34.068 INFO:teuthology.orchestra.run.smithi044.stdout: pg 3.1a is active+undersized+degraded, acting [4,1] 2024-09-17T07:22:34.068 INFO:teuthology.orchestra.run.smithi044.stdout: pg 3.1e is active+undersized+degraded, acting [3,6] 2024-09-17T07:22:34.068 INFO:teuthology.orchestra.run.smithi044.stdout: pg 3.1f is active+undersized+degraded, acting [0,5] 2024-09-17T07:22:34.068 INFO:teuthology.orchestra.run.smithi044.stdout: pg 4.d is active+undersized+degraded, acting [4,1] 2024-09-17T07:22:34.068 INFO:teuthology.orchestra.run.smithi044.stdout: pg 4.14 is active+undersized+degraded, acting [3,7] 2024-09-17T07:22:34.117 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:33 smithi179 ceph-mon[50085]: from='client.54505 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:22:34.117 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:33 smithi179 ceph-mon[50085]: from='client.54511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:22:34.117 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:33 smithi179 ceph-mon[50085]: pgmap v105: 161 pgs: 35 active+undersized, 13 active+undersized+degraded, 113 active+clean; 604 KiB data, 164 MiB used, 715 GiB / 715 GiB avail; 88/999 objects degraded (8.809%) 2024-09-17T07:22:34.117 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:33 smithi179 ceph-mon[50085]: from='client.44634 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:22:34.117 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:33 smithi179 ceph-mon[50085]: osdmap e91: 8 total, 8 up, 8 in 2024-09-17T07:22:34.117 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:33 smithi179 ceph-mon[50085]: from='client.54523 172.21.15.44:0/2673971454' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:22:34.117 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:33 smithi179 ceph-mon[50085]: Health check update: Degraded data redundancy: 88/999 objects degraded (8.809%), 13 pgs degraded (PG_DEGRADED) 2024-09-17T07:22:34.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:34 smithi044 ceph-mon[58191]: from='client.44646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:22:34.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:34 smithi044 ceph-mon[58191]: from='client.54535 172.21.15.44:0/375645241' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:22:34.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:34 smithi044 ceph-mon[60356]: from='client.44646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:22:34.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:34 smithi044 ceph-mon[60356]: from='client.54535 172.21.15.44:0/375645241' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:22:35.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:34 smithi179 ceph-mon[50085]: from='client.44646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:22:35.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:34 smithi179 ceph-mon[50085]: from='client.54535 172.21.15.44:0/375645241' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:22:36.015 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:35 smithi179 ceph-mon[50085]: pgmap v107: 161 pgs: 32 active+undersized, 10 active+undersized+degraded, 119 active+clean; 604 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 60/999 objects degraded (6.006%) 2024-09-17T07:22:36.015 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:35 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:36.015 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:35 smithi179 ceph-mon[50085]: from='client.54553 172.21.15.44:0/4136917136' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:22:36.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:35 smithi044 ceph-mon[58191]: pgmap v107: 161 pgs: 32 active+undersized, 10 active+undersized+degraded, 119 active+clean; 604 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 60/999 objects degraded (6.006%) 2024-09-17T07:22:36.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:35 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:36.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:35 smithi044 ceph-mon[58191]: from='client.54553 172.21.15.44:0/4136917136' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:22:36.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:35 smithi044 ceph-mon[60356]: pgmap v107: 161 pgs: 32 active+undersized, 10 active+undersized+degraded, 119 active+clean; 604 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 60/999 objects degraded (6.006%) 2024-09-17T07:22:36.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:35 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:36.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:35 smithi044 ceph-mon[60356]: from='client.54553 172.21.15.44:0/4136917136' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:22:37.062 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:36 smithi044 ceph-mon[58191]: from='client.44676 172.21.15.44:0/4025718360' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/1355368272"}]: access denied 2024-09-17T07:22:37.062 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:36 smithi044 ceph-mon[60356]: from='client.44676 172.21.15.44:0/4025718360' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/1355368272"}]: access denied 2024-09-17T07:22:37.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:36 smithi179 ceph-mon[50085]: from='client.44676 172.21.15.44:0/4025718360' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/1355368272"}]: access denied 2024-09-17T07:22:37.900 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:37 smithi044 ceph-mon[58191]: pgmap v108: 161 pgs: 161 active+clean; 604 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-17T07:22:37.901 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:37 smithi044 ceph-mon[58191]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 60/999 objects degraded (6.006%), 10 pgs degraded) 2024-09-17T07:22:37.901 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:37 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:37.901 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:37 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:37.901 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:37 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:22:37.901 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:37 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:22:37.901 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:37 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:37.901 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:37 smithi044 ceph-mon[60356]: pgmap v108: 161 pgs: 161 active+clean; 604 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-17T07:22:37.901 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:37 smithi044 ceph-mon[60356]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 60/999 objects degraded (6.006%), 10 pgs degraded) 2024-09-17T07:22:37.901 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:37 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:37.901 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:37 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:37.901 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:37 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:22:37.902 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:37 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:22:37.902 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:37 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:38.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:37 smithi179 ceph-mon[50085]: pgmap v108: 161 pgs: 161 active+clean; 604 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-17T07:22:38.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:37 smithi179 ceph-mon[50085]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 60/999 objects degraded (6.006%), 10 pgs degraded) 2024-09-17T07:22:38.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:37 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:38.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:37 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:38.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:37 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:22:38.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:37 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:22:38.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:37 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:39.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:38 smithi044 ceph-mon[60356]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:22:39.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:38 smithi044 ceph-mon[60356]: pgmap v109: 161 pgs: 161 active+clean; 604 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 148 B/s rd, 0 op/s 2024-09-17T07:22:39.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:38 smithi044 ceph-mon[60356]: pgmap v110: 161 pgs: 161 active+clean; 604 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 187 B/s rd, 0 op/s 2024-09-17T07:22:39.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:38 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:22:39.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:38 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:22:39.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:38 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:22:39.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:38 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:22:39.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:38 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-17T07:22:39.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:38 smithi044 ceph-mon[60356]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-17T07:22:39.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:38 smithi044 ceph-mon[60356]: Upgrade: osd.3 is safe to restart 2024-09-17T07:22:39.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:38 smithi044 ceph-mon[58191]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:22:39.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:38 smithi044 ceph-mon[58191]: pgmap v109: 161 pgs: 161 active+clean; 604 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 148 B/s rd, 0 op/s 2024-09-17T07:22:39.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:38 smithi044 ceph-mon[58191]: pgmap v110: 161 pgs: 161 active+clean; 604 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 187 B/s rd, 0 op/s 2024-09-17T07:22:39.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:38 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:22:39.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:38 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:22:39.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:38 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:22:39.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:38 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:22:39.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:38 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-17T07:22:39.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:38 smithi044 ceph-mon[58191]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-17T07:22:39.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:38 smithi044 ceph-mon[58191]: Upgrade: osd.3 is safe to restart 2024-09-17T07:22:39.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:38 smithi179 ceph-mon[50085]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:22:39.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:38 smithi179 ceph-mon[50085]: pgmap v109: 161 pgs: 161 active+clean; 604 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 148 B/s rd, 0 op/s 2024-09-17T07:22:39.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:38 smithi179 ceph-mon[50085]: pgmap v110: 161 pgs: 161 active+clean; 604 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 187 B/s rd, 0 op/s 2024-09-17T07:22:39.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:38 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:22:39.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:38 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:22:39.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:38 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:22:39.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:38 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:22:39.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:38 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-17T07:22:39.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:38 smithi179 ceph-mon[50085]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-17T07:22:39.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:38 smithi179 ceph-mon[50085]: Upgrade: osd.3 is safe to restart 2024-09-17T07:22:40.506 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:40 smithi044 ceph-mon[58191]: Upgrade: Updating osd.3 2024-09-17T07:22:40.506 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:40 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:40.507 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:40 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-17T07:22:40.507 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:40 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:22:40.507 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:40 smithi044 ceph-mon[58191]: Deploying daemon osd.3 on smithi044 2024-09-17T07:22:40.507 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:40 smithi044 ceph-mon[58191]: pgmap v111: 161 pgs: 161 active+clean; 604 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 144 B/s rd, 0 op/s 2024-09-17T07:22:40.507 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:40 smithi044 ceph-mon[60356]: Upgrade: Updating osd.3 2024-09-17T07:22:40.507 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:40 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:40.507 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:40 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-17T07:22:40.507 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:40 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:22:40.507 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:40 smithi044 ceph-mon[60356]: Deploying daemon osd.3 on smithi044 2024-09-17T07:22:40.507 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:40 smithi044 ceph-mon[60356]: pgmap v111: 161 pgs: 161 active+clean; 604 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 144 B/s rd, 0 op/s 2024-09-17T07:22:40.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:40 smithi179 ceph-mon[50085]: Upgrade: Updating osd.3 2024-09-17T07:22:40.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:40 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:40.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:40 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-17T07:22:40.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:40 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:22:40.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:40 smithi179 ceph-mon[50085]: Deploying daemon osd.3 on smithi044 2024-09-17T07:22:40.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:40 smithi179 ceph-mon[50085]: pgmap v111: 161 pgs: 161 active+clean; 604 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 144 B/s rd, 0 op/s 2024-09-17T07:22:42.270 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:42 smithi044 systemd[1]: Stopping Ceph osd.3 for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:22:42.574 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.3[36511]: debug 2024-09-17T07:22:42.267+0000 7fe78fd26700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-17T07:22:42.575 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.3[36511]: debug 2024-09-17T07:22:42.268+0000 7fe78fd26700 -1 osd.3 91 *** Got signal Terminated *** 2024-09-17T07:22:42.575 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.3[36511]: debug 2024-09-17T07:22:42.268+0000 7fe78fd26700 -1 osd.3 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T07:22:42.575 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:42 smithi044 podman[73308]: 2024-09-17 07:22:42.310394123 +0000 UTC m=+0.143691714 container died 2b7d623395a75bac0f4b14a62081bdd26f4dad851c4e3c4ccd521e2722bec957 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.3, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, ceph=True, org.label-schema.build-date=20201204, CEPH_POINT_RELEASE=-16.2.0, GIT_CLEAN=True, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Dimitri Savineau ) 2024-09-17T07:22:42.937 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:42 smithi044 podman[73308]: 2024-09-17 07:22:42.685869616 +0000 UTC m=+0.519167210 container cleanup 2b7d623395a75bac0f4b14a62081bdd26f4dad851c4e3c4ccd521e2722bec957 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.3, org.label-schema.build-date=20201204, GIT_CLEAN=True, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Base Image, org.label-schema.license=GPLv2, RELEASE=HEAD, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-16.2.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Dimitri Savineau ) 2024-09-17T07:22:42.937 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:42 smithi044 bash[73308]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.3 2024-09-17T07:22:42.938 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[60356]: pgmap v112: 161 pgs: 161 active+clean; 604 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 120 B/s rd, 0 op/s 2024-09-17T07:22:42.938 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[60356]: osd.3 reported immediately failed by osd.4 2024-09-17T07:22:42.938 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[60356]: osd.3 failed (root=default,host=smithi044) (connection refused reported by osd.4) 2024-09-17T07:22:42.938 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[60356]: osd.3 reported immediately failed by osd.0 2024-09-17T07:22:42.938 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[60356]: osd.3 reported immediately failed by osd.1 2024-09-17T07:22:42.938 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[60356]: osd.3 reported immediately failed by osd.1 2024-09-17T07:22:42.938 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[60356]: osd.3 reported immediately failed by osd.2 2024-09-17T07:22:42.938 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[60356]: osd.3 reported immediately failed by osd.6 2024-09-17T07:22:42.938 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[60356]: osd.3 reported immediately failed by osd.5 2024-09-17T07:22:42.938 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[60356]: osd.3 reported immediately failed by osd.4 2024-09-17T07:22:42.938 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[60356]: osd.3 reported immediately failed by osd.0 2024-09-17T07:22:42.938 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[60356]: osd.3 reported immediately failed by osd.2 2024-09-17T07:22:42.938 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[60356]: osd.3 reported immediately failed by osd.7 2024-09-17T07:22:42.939 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[60356]: osd.3 reported immediately failed by osd.6 2024-09-17T07:22:42.939 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[60356]: osd.3 reported immediately failed by osd.5 2024-09-17T07:22:42.939 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[60356]: osd.3 reported immediately failed by osd.4 2024-09-17T07:22:42.939 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[60356]: osd.3 reported immediately failed by osd.0 2024-09-17T07:22:42.939 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[60356]: osd.3 reported immediately failed by osd.7 2024-09-17T07:22:42.939 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[60356]: osd.3 reported immediately failed by osd.4 2024-09-17T07:22:42.939 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[60356]: osd.3 reported immediately failed by osd.0 2024-09-17T07:22:42.939 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[60356]: osd.3 reported immediately failed by osd.5 2024-09-17T07:22:42.939 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[58191]: pgmap v112: 161 pgs: 161 active+clean; 604 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 120 B/s rd, 0 op/s 2024-09-17T07:22:42.939 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[58191]: osd.3 reported immediately failed by osd.4 2024-09-17T07:22:42.939 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[58191]: osd.3 failed (root=default,host=smithi044) (connection refused reported by osd.4) 2024-09-17T07:22:42.940 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[58191]: osd.3 reported immediately failed by osd.0 2024-09-17T07:22:42.940 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[58191]: osd.3 reported immediately failed by osd.1 2024-09-17T07:22:42.940 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[58191]: osd.3 reported immediately failed by osd.1 2024-09-17T07:22:42.940 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[58191]: osd.3 reported immediately failed by osd.2 2024-09-17T07:22:42.940 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[58191]: osd.3 reported immediately failed by osd.6 2024-09-17T07:22:42.940 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[58191]: osd.3 reported immediately failed by osd.5 2024-09-17T07:22:42.940 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[58191]: osd.3 reported immediately failed by osd.4 2024-09-17T07:22:42.940 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[58191]: osd.3 reported immediately failed by osd.0 2024-09-17T07:22:42.940 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[58191]: osd.3 reported immediately failed by osd.2 2024-09-17T07:22:42.940 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[58191]: osd.3 reported immediately failed by osd.7 2024-09-17T07:22:42.940 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[58191]: osd.3 reported immediately failed by osd.6 2024-09-17T07:22:42.940 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[58191]: osd.3 reported immediately failed by osd.5 2024-09-17T07:22:42.940 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[58191]: osd.3 reported immediately failed by osd.4 2024-09-17T07:22:42.941 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[58191]: osd.3 reported immediately failed by osd.0 2024-09-17T07:22:42.941 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[58191]: osd.3 reported immediately failed by osd.7 2024-09-17T07:22:42.941 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[58191]: osd.3 reported immediately failed by osd.4 2024-09-17T07:22:42.941 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[58191]: osd.3 reported immediately failed by osd.0 2024-09-17T07:22:42.941 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:42 smithi044 ceph-mon[58191]: osd.3 reported immediately failed by osd.5 2024-09-17T07:22:43.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:42 smithi179 ceph-mon[50085]: pgmap v112: 161 pgs: 161 active+clean; 604 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 120 B/s rd, 0 op/s 2024-09-17T07:22:43.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:42 smithi179 ceph-mon[50085]: osd.3 reported immediately failed by osd.4 2024-09-17T07:22:43.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:42 smithi179 ceph-mon[50085]: osd.3 failed (root=default,host=smithi044) (connection refused reported by osd.4) 2024-09-17T07:22:43.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:42 smithi179 ceph-mon[50085]: osd.3 reported immediately failed by osd.0 2024-09-17T07:22:43.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:42 smithi179 ceph-mon[50085]: osd.3 reported immediately failed by osd.1 2024-09-17T07:22:43.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:42 smithi179 ceph-mon[50085]: osd.3 reported immediately failed by osd.1 2024-09-17T07:22:43.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:42 smithi179 ceph-mon[50085]: osd.3 reported immediately failed by osd.2 2024-09-17T07:22:43.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:42 smithi179 ceph-mon[50085]: osd.3 reported immediately failed by osd.6 2024-09-17T07:22:43.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:42 smithi179 ceph-mon[50085]: osd.3 reported immediately failed by osd.5 2024-09-17T07:22:43.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:42 smithi179 ceph-mon[50085]: osd.3 reported immediately failed by osd.4 2024-09-17T07:22:43.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:42 smithi179 ceph-mon[50085]: osd.3 reported immediately failed by osd.0 2024-09-17T07:22:43.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:42 smithi179 ceph-mon[50085]: osd.3 reported immediately failed by osd.2 2024-09-17T07:22:43.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:42 smithi179 ceph-mon[50085]: osd.3 reported immediately failed by osd.7 2024-09-17T07:22:43.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:42 smithi179 ceph-mon[50085]: osd.3 reported immediately failed by osd.6 2024-09-17T07:22:43.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:42 smithi179 ceph-mon[50085]: osd.3 reported immediately failed by osd.5 2024-09-17T07:22:43.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:42 smithi179 ceph-mon[50085]: osd.3 reported immediately failed by osd.4 2024-09-17T07:22:43.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:42 smithi179 ceph-mon[50085]: osd.3 reported immediately failed by osd.0 2024-09-17T07:22:43.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:42 smithi179 ceph-mon[50085]: osd.3 reported immediately failed by osd.7 2024-09-17T07:22:43.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:42 smithi179 ceph-mon[50085]: osd.3 reported immediately failed by osd.4 2024-09-17T07:22:43.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:42 smithi179 ceph-mon[50085]: osd.3 reported immediately failed by osd.0 2024-09-17T07:22:43.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:42 smithi179 ceph-mon[50085]: osd.3 reported immediately failed by osd.5 2024-09-17T07:22:43.325 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:42 smithi044 podman[73321]: 2024-09-17 07:22:42.937413343 +0000 UTC m=+0.626200294 container remove 2b7d623395a75bac0f4b14a62081bdd26f4dad851c4e3c4ccd521e2722bec957 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.3, maintainer=Dimitri Savineau , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-16.2.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, RELEASE=HEAD, org.label-schema.build-date=20201204, org.label-schema.schema-version=1.0) 2024-09-17T07:22:43.709 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:43 smithi044 podman[73425]: 2024-09-17 07:22:43.321929046 +0000 UTC m=+0.016446653 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:22:43.710 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:43 smithi044 podman[73425]: 2024-09-17 07:22:43.441620244 +0000 UTC m=+0.136137849 container create 0c7898bef2b209b577de41d2c77826fe2bc9e0a51f670651edd3cbc624cdb3ad (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.license=GPLv2) 2024-09-17T07:22:43.710 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:43 smithi044 podman[73425]: 2024-09-17 07:22:43.649058116 +0000 UTC m=+0.343575721 container init 0c7898bef2b209b577de41d2c77826fe2bc9e0a51f670651edd3cbc624cdb3ad (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3-deactivate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-17T07:22:43.710 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:43 smithi044 podman[73425]: 2024-09-17 07:22:43.652543029 +0000 UTC m=+0.347060627 container start 0c7898bef2b209b577de41d2c77826fe2bc9e0a51f670651edd3cbc624cdb3ad (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-17T07:22:44.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:43 smithi044 ceph-mon[58191]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T07:22:44.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:43 smithi044 ceph-mon[58191]: osdmap e92: 8 total, 7 up, 8 in 2024-09-17T07:22:44.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:43 smithi044 ceph-mon[60356]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T07:22:44.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:43 smithi044 ceph-mon[60356]: osdmap e92: 8 total, 7 up, 8 in 2024-09-17T07:22:44.075 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:43 smithi044 podman[73425]: 2024-09-17 07:22:43.74103949 +0000 UTC m=+0.435557105 container attach 0c7898bef2b209b577de41d2c77826fe2bc9e0a51f670651edd3cbc624cdb3ad (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, ceph=True) 2024-09-17T07:22:44.075 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:43 smithi044 podman[73425]: 2024-09-17 07:22:43.8558831 +0000 UTC m=+0.550400698 container died 0c7898bef2b209b577de41d2c77826fe2bc9e0a51f670651edd3cbc624cdb3ad (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3-deactivate, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-17T07:22:44.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:43 smithi179 ceph-mon[50085]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T07:22:44.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:43 smithi179 ceph-mon[50085]: osdmap e92: 8 total, 7 up, 8 in 2024-09-17T07:22:44.414 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:44 smithi044 podman[73448]: 2024-09-17 07:22:44.411497275 +0000 UTC m=+0.549581133 container remove 0c7898bef2b209b577de41d2c77826fe2bc9e0a51f670651edd3cbc624cdb3ad (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-17T07:22:44.710 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:44 smithi044 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.3.service: Deactivated successfully. 2024-09-17T07:22:44.711 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:44 smithi044 systemd[1]: Stopped Ceph osd.3 for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:22:44.711 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:44 smithi044 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.3.service: Consumed 1.154s CPU time. 2024-09-17T07:22:44.711 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:44 smithi044 systemd[1]: Starting Ceph osd.3 for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:22:45.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:44 smithi044 ceph-mon[58191]: pgmap v114: 161 pgs: 27 stale+active+clean, 134 active+clean; 604 KiB data, 169 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:22:45.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:44 smithi044 ceph-mon[58191]: osdmap e93: 8 total, 7 up, 8 in 2024-09-17T07:22:45.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:44 smithi044 ceph-mon[60356]: pgmap v114: 161 pgs: 27 stale+active+clean, 134 active+clean; 604 KiB data, 169 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:22:45.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:44 smithi044 ceph-mon[60356]: osdmap e93: 8 total, 7 up, 8 in 2024-09-17T07:22:45.075 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:44 smithi044 podman[73548]: 2024-09-17 07:22:44.763212859 +0000 UTC m=+0.016365054 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:22:45.075 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:44 smithi044 podman[73548]: 2024-09-17 07:22:44.870874275 +0000 UTC m=+0.124026518 container create 97dc30a4258be0ccc47f1437b2755c3a54626ba20c08962358bae304b9973092 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3-activate, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b) 2024-09-17T07:22:45.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:44 smithi179 ceph-mon[50085]: pgmap v114: 161 pgs: 27 stale+active+clean, 134 active+clean; 604 KiB data, 169 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:22:45.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:44 smithi179 ceph-mon[50085]: osdmap e93: 8 total, 7 up, 8 in 2024-09-17T07:22:45.537 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:45 smithi044 podman[73548]: 2024-09-17 07:22:45.22349742 +0000 UTC m=+0.476649626 container init 97dc30a4258be0ccc47f1437b2755c3a54626ba20c08962358bae304b9973092 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3-activate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-17T07:22:45.537 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:45 smithi044 podman[73548]: 2024-09-17 07:22:45.226975413 +0000 UTC m=+0.480127608 container start 97dc30a4258be0ccc47f1437b2755c3a54626ba20c08962358bae304b9973092 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3-activate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T07:22:45.537 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:45 smithi044 podman[73548]: 2024-09-17 07:22:45.323490101 +0000 UTC m=+0.576642484 container attach 97dc30a4258be0ccc47f1437b2755c3a54626ba20c08962358bae304b9973092 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3-activate, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD) 2024-09-17T07:22:45.825 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3-activate[73562]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-17T07:22:45.825 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:45 smithi044 bash[73548]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-17T07:22:45.825 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3-activate[73562]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-17T07:22:45.825 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:45 smithi044 bash[73548]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-17T07:22:45.825 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3-activate[73562]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-17T07:22:45.825 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:45 smithi044 bash[73548]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-17T07:22:45.825 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3-activate[73562]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-17T07:22:45.825 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:45 smithi044 bash[73548]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-17T07:22:45.825 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3-activate[73562]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-17T07:22:45.825 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:45 smithi044 bash[73548]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-17T07:22:45.825 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3-activate[73562]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-17T07:22:45.826 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:45 smithi044 bash[73548]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-17T07:22:45.826 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:45 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3-activate[73562]: --> ceph-volume raw activate successful for osd ID: 3 2024-09-17T07:22:45.826 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:45 smithi044 bash[73548]: --> ceph-volume raw activate successful for osd ID: 3 2024-09-17T07:22:45.826 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:45 smithi044 podman[73548]: 2024-09-17 07:22:45.585934577 +0000 UTC m=+0.839086771 container died 97dc30a4258be0ccc47f1437b2755c3a54626ba20c08962358bae304b9973092 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3-activate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b) 2024-09-17T07:22:46.278 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:46 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:46.278 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:46 smithi044 ceph-mon[58191]: pgmap v116: 161 pgs: 9 active+undersized, 13 peering, 16 stale+active+clean, 5 active+undersized+degraded, 118 active+clean; 604 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 46/999 objects degraded (4.605%) 2024-09-17T07:22:46.278 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:46 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:46.278 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:46 smithi044 ceph-mon[60356]: pgmap v116: 161 pgs: 9 active+undersized, 13 peering, 16 stale+active+clean, 5 active+undersized+degraded, 118 active+clean; 604 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 46/999 objects degraded (4.605%) 2024-09-17T07:22:46.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:46 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:46.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:46 smithi179 ceph-mon[50085]: pgmap v116: 161 pgs: 9 active+undersized, 13 peering, 16 stale+active+clean, 5 active+undersized+degraded, 118 active+clean; 604 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 46/999 objects degraded (4.605%) 2024-09-17T07:22:46.574 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:46 smithi044 podman[73692]: 2024-09-17 07:22:46.277988275 +0000 UTC m=+0.685702923 container remove 97dc30a4258be0ccc47f1437b2755c3a54626ba20c08962358bae304b9973092 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3-activate, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-17T07:22:46.574 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:46 smithi044 podman[73752]: 2024-09-17 07:22:46.393956913 +0000 UTC m=+0.016637745 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:22:46.839 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:46 smithi044 podman[73752]: 2024-09-17 07:22:46.578874485 +0000 UTC m=+0.201555322 container create b4323c928f39eccf3759a07f1e9cd5db2584c879de4d46cfebb699fe9d041f16 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T07:22:46.839 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:46 smithi044 podman[73752]: 2024-09-17 07:22:46.814806893 +0000 UTC m=+0.437487727 container init b4323c928f39eccf3759a07f1e9cd5db2584c879de4d46cfebb699fe9d041f16 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-17T07:22:46.839 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:46 smithi044 podman[73752]: 2024-09-17 07:22:46.81825153 +0000 UTC m=+0.440932362 container start b4323c928f39eccf3759a07f1e9cd5db2584c879de4d46cfebb699fe9d041f16 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-17T07:22:47.090 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:46 smithi044 ceph-osd[73772]: -- 172.21.15.44:0/3689423853 <== mon.1 v2:172.21.15.44:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5589fa2eb1e0 con 0x5589fb06c400 2024-09-17T07:22:47.090 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:46 smithi044 bash[73752]: b4323c928f39eccf3759a07f1e9cd5db2584c879de4d46cfebb699fe9d041f16 2024-09-17T07:22:47.090 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:46 smithi044 systemd[1]: Started Ceph osd.3 for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:22:47.363 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:47 smithi044 ceph-mon[58191]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-17T07:22:47.363 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:47 smithi044 ceph-mon[58191]: Health check failed: Degraded data redundancy: 46/999 objects degraded (4.605%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T07:22:47.363 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:47 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:47.363 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:47 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:47.363 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:47 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:22:47.363 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:47 smithi044 ceph-mon[60356]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-17T07:22:47.363 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:47 smithi044 ceph-mon[60356]: Health check failed: Degraded data redundancy: 46/999 objects degraded (4.605%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T07:22:47.363 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:47 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:47.364 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:47 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:47.364 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:47 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:22:47.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:47 smithi179 ceph-mon[50085]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-17T07:22:47.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:47 smithi179 ceph-mon[50085]: Health check failed: Degraded data redundancy: 46/999 objects degraded (4.605%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T07:22:47.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:47 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:47.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:47 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:47.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:47 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:22:47.620 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:47 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3[73768]: 2024-09-17T07:22:47.359+0000 7fbfe0fb8640 -1 Falling back to public interface 2024-09-17T07:22:48.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:48 smithi044 ceph-mon[58191]: pgmap v117: 161 pgs: 36 active+undersized, 13 peering, 17 active+undersized+degraded, 95 active+clean; 604 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 108/999 objects degraded (10.811%) 2024-09-17T07:22:48.333 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:48 smithi044 ceph-mon[60356]: pgmap v117: 161 pgs: 36 active+undersized, 13 peering, 17 active+undersized+degraded, 95 active+clean; 604 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 108/999 objects degraded (10.811%) 2024-09-17T07:22:48.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:48 smithi179 ceph-mon[50085]: pgmap v117: 161 pgs: 36 active+undersized, 13 peering, 17 active+undersized+degraded, 95 active+clean; 604 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 108/999 objects degraded (10.811%) 2024-09-17T07:22:49.939 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:49 smithi044 ceph-mon[58191]: from='client.54580 172.21.15.44:0/3982164599' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:22:49.940 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:49 smithi044 ceph-mon[60356]: from='client.54580 172.21.15.44:0/3982164599' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:22:50.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:49 smithi179 ceph-mon[50085]: from='client.54580 172.21.15.44:0/3982164599' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:22:50.574 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3[73768]: 2024-09-17T07:22:50.202+0000 7fbfe0fb8640 -1 osd.3 0 read_superblock omap replica is missing. 2024-09-17T07:22:50.575 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:50 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3[73768]: 2024-09-17T07:22:50.229+0000 7fbfe0fb8640 -1 osd.3 91 log_to_monitors true 2024-09-17T07:22:51.011 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:50 smithi044 ceph-mon[60356]: pgmap v118: 161 pgs: 45 active+undersized, 21 active+undersized+degraded, 95 active+clean; 604 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 133/999 objects degraded (13.313%) 2024-09-17T07:22:51.011 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:50 smithi044 ceph-mon[60356]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs peering) 2024-09-17T07:22:51.011 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:50 smithi044 ceph-mon[60356]: from='client.54586 172.21.15.44:0/1350317956' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:22:51.011 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:50 smithi044 ceph-mon[60356]: from='osd.3 [v2:172.21.15.44:6826/3921361135,v1:172.21.15.44:6827/3921361135]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-17T07:22:51.011 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:50 smithi044 ceph-mon[60356]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-17T07:22:51.011 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:50 smithi044 ceph-mon[58191]: pgmap v118: 161 pgs: 45 active+undersized, 21 active+undersized+degraded, 95 active+clean; 604 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 133/999 objects degraded (13.313%) 2024-09-17T07:22:51.011 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:50 smithi044 ceph-mon[58191]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs peering) 2024-09-17T07:22:51.012 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:50 smithi044 ceph-mon[58191]: from='client.54586 172.21.15.44:0/1350317956' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:22:51.012 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:50 smithi044 ceph-mon[58191]: from='osd.3 [v2:172.21.15.44:6826/3921361135,v1:172.21.15.44:6827/3921361135]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-17T07:22:51.012 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:50 smithi044 ceph-mon[58191]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-17T07:22:51.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:50 smithi179 ceph-mon[50085]: pgmap v118: 161 pgs: 45 active+undersized, 21 active+undersized+degraded, 95 active+clean; 604 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 133/999 objects degraded (13.313%) 2024-09-17T07:22:51.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:50 smithi179 ceph-mon[50085]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs peering) 2024-09-17T07:22:51.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:50 smithi179 ceph-mon[50085]: from='client.54586 172.21.15.44:0/1350317956' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:22:51.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:50 smithi179 ceph-mon[50085]: from='osd.3 [v2:172.21.15.44:6826/3921361135,v1:172.21.15.44:6827/3921361135]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-17T07:22:51.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:50 smithi179 ceph-mon[50085]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-17T07:22:51.324 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:22:51 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3[73768]: 2024-09-17T07:22:51.152+0000 7fbfd4d5a640 -1 osd.3 91 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T07:22:52.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:51 smithi044 ceph-mon[58191]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-17T07:22:52.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:51 smithi044 ceph-mon[58191]: osdmap e94: 8 total, 7 up, 8 in 2024-09-17T07:22:52.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:51 smithi044 ceph-mon[58191]: from='osd.3 [v2:172.21.15.44:6826/3921361135,v1:172.21.15.44:6827/3921361135]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-09-17T07:22:52.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:51 smithi044 ceph-mon[58191]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-09-17T07:22:52.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:51 smithi044 ceph-mon[58191]: from='osd.3 ' entity='osd.3' 2024-09-17T07:22:52.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:51 smithi044 ceph-mon[60356]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-17T07:22:52.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:51 smithi044 ceph-mon[60356]: osdmap e94: 8 total, 7 up, 8 in 2024-09-17T07:22:52.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:51 smithi044 ceph-mon[60356]: from='osd.3 [v2:172.21.15.44:6826/3921361135,v1:172.21.15.44:6827/3921361135]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-09-17T07:22:52.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:51 smithi044 ceph-mon[60356]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-09-17T07:22:52.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:51 smithi044 ceph-mon[60356]: from='osd.3 ' entity='osd.3' 2024-09-17T07:22:52.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:51 smithi179 ceph-mon[50085]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-17T07:22:52.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:51 smithi179 ceph-mon[50085]: osdmap e94: 8 total, 7 up, 8 in 2024-09-17T07:22:52.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:51 smithi179 ceph-mon[50085]: from='osd.3 [v2:172.21.15.44:6826/3921361135,v1:172.21.15.44:6827/3921361135]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-09-17T07:22:52.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:51 smithi179 ceph-mon[50085]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-09-17T07:22:52.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:51 smithi179 ceph-mon[50085]: from='osd.3 ' entity='osd.3' 2024-09-17T07:22:53.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:52 smithi044 ceph-mon[60356]: pgmap v120: 161 pgs: 45 active+undersized, 21 active+undersized+degraded, 95 active+clean; 604 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 133/999 objects degraded (13.313%) 2024-09-17T07:22:53.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:52 smithi044 ceph-mon[60356]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T07:22:53.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:52 smithi044 ceph-mon[60356]: osd.3 [v2:172.21.15.44:6826/3921361135,v1:172.21.15.44:6827/3921361135] boot 2024-09-17T07:22:53.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:52 smithi044 ceph-mon[60356]: osdmap e95: 8 total, 8 up, 8 in 2024-09-17T07:22:53.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:52 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:22:53.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:52 smithi044 ceph-mon[58191]: pgmap v120: 161 pgs: 45 active+undersized, 21 active+undersized+degraded, 95 active+clean; 604 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 133/999 objects degraded (13.313%) 2024-09-17T07:22:53.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:52 smithi044 ceph-mon[58191]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T07:22:53.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:52 smithi044 ceph-mon[58191]: osd.3 [v2:172.21.15.44:6826/3921361135,v1:172.21.15.44:6827/3921361135] boot 2024-09-17T07:22:53.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:52 smithi044 ceph-mon[58191]: osdmap e95: 8 total, 8 up, 8 in 2024-09-17T07:22:53.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:52 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:22:53.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:52 smithi179 ceph-mon[50085]: pgmap v120: 161 pgs: 45 active+undersized, 21 active+undersized+degraded, 95 active+clean; 604 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 133/999 objects degraded (13.313%) 2024-09-17T07:22:53.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:52 smithi179 ceph-mon[50085]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T07:22:53.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:52 smithi179 ceph-mon[50085]: osd.3 [v2:172.21.15.44:6826/3921361135,v1:172.21.15.44:6827/3921361135] boot 2024-09-17T07:22:53.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:52 smithi179 ceph-mon[50085]: osdmap e95: 8 total, 8 up, 8 in 2024-09-17T07:22:53.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:52 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T07:22:54.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:54 smithi044 ceph-mon[58191]: osdmap e96: 8 total, 8 up, 8 in 2024-09-17T07:22:54.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:54 smithi044 ceph-mon[58191]: Health check update: Degraded data redundancy: 133/999 objects degraded (13.313%), 21 pgs degraded (PG_DEGRADED) 2024-09-17T07:22:54.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:54 smithi044 ceph-mon[58191]: pgmap v123: 161 pgs: 45 active+undersized, 21 active+undersized+degraded, 95 active+clean; 604 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 133/999 objects degraded (13.313%) 2024-09-17T07:22:54.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:54 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:54.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:54 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:54.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:54 smithi044 ceph-mon[60356]: osdmap e96: 8 total, 8 up, 8 in 2024-09-17T07:22:54.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:54 smithi044 ceph-mon[60356]: Health check update: Degraded data redundancy: 133/999 objects degraded (13.313%), 21 pgs degraded (PG_DEGRADED) 2024-09-17T07:22:54.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:54 smithi044 ceph-mon[60356]: pgmap v123: 161 pgs: 45 active+undersized, 21 active+undersized+degraded, 95 active+clean; 604 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 133/999 objects degraded (13.313%) 2024-09-17T07:22:54.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:54 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:54.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:54 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:54.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:54 smithi179 ceph-mon[50085]: osdmap e96: 8 total, 8 up, 8 in 2024-09-17T07:22:54.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:54 smithi179 ceph-mon[50085]: Health check update: Degraded data redundancy: 133/999 objects degraded (13.313%), 21 pgs degraded (PG_DEGRADED) 2024-09-17T07:22:54.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:54 smithi179 ceph-mon[50085]: pgmap v123: 161 pgs: 45 active+undersized, 21 active+undersized+degraded, 95 active+clean; 604 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 133/999 objects degraded (13.313%) 2024-09-17T07:22:54.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:54 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:54.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:54 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:55.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:55 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:55.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:55 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:55.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:55 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:55.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:55 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:55.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:55 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:55.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:55 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:55.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:55 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:55.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:55 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:55.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:55 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:56.694 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:56 smithi044 ceph-mon[58191]: pgmap v124: 161 pgs: 35 active+undersized, 15 active+undersized+degraded, 111 active+clean; 604 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 91/999 objects degraded (9.109%) 2024-09-17T07:22:56.694 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:56 smithi044 ceph-mon[60356]: pgmap v124: 161 pgs: 35 active+undersized, 15 active+undersized+degraded, 111 active+clean; 604 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 91/999 objects degraded (9.109%) 2024-09-17T07:22:56.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:56 smithi179 ceph-mon[50085]: pgmap v124: 161 pgs: 35 active+undersized, 15 active+undersized+degraded, 111 active+clean; 604 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 91/999 objects degraded (9.109%) 2024-09-17T07:22:57.991 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:57 smithi044 ceph-mon[58191]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 91/999 objects degraded (9.109%), 15 pgs degraded) 2024-09-17T07:22:57.991 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:57 smithi044 ceph-mon[60356]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 91/999 objects degraded (9.109%), 15 pgs degraded) 2024-09-17T07:22:58.113 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:57 smithi179 ceph-mon[50085]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 91/999 objects degraded (9.109%), 15 pgs degraded) 2024-09-17T07:22:59.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:58 smithi044 ceph-mon[58191]: pgmap v125: 161 pgs: 161 active+clean; 604 KiB data, 194 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:22:59.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:58 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:59.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:58 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:59.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:58 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:22:59.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:58 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:22:59.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:58 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:59.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:58 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:22:59.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:58 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:22:59.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:58 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:22:59.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:58 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:22:59.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:58 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-17T07:22:59.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:58 smithi044 ceph-mon[60356]: pgmap v125: 161 pgs: 161 active+clean; 604 KiB data, 194 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:22:59.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:58 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:59.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:58 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:59.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:58 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:22:59.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:58 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:22:59.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:58 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:59.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:58 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:22:59.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:58 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:22:59.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:58 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:22:59.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:58 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:22:59.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:58 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-17T07:22:59.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:58 smithi179 ceph-mon[50085]: pgmap v125: 161 pgs: 161 active+clean; 604 KiB data, 194 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:22:59.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:58 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:59.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:58 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:59.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:58 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:22:59.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:58 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:22:59.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:58 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:59.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:58 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:22:59.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:58 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:22:59.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:58 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:22:59.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:58 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:22:59.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:58 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-17T07:22:59.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:59 smithi179 ceph-mon[50085]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:22:59.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:59 smithi179 ceph-mon[50085]: pgmap v126: 161 pgs: 161 active+clean; 604 KiB data, 194 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:22:59.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:59 smithi179 ceph-mon[50085]: pgmap v127: 161 pgs: 161 active+clean; 604 KiB data, 194 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:22:59.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:59 smithi179 ceph-mon[50085]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-17T07:22:59.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:59 smithi179 ceph-mon[50085]: Upgrade: osd.4 is safe to restart 2024-09-17T07:22:59.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:59 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:22:59.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:59 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-17T07:22:59.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:22:59 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:23:00.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:59 smithi044 ceph-mon[58191]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:23:00.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:59 smithi044 ceph-mon[58191]: pgmap v126: 161 pgs: 161 active+clean; 604 KiB data, 194 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:23:00.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:59 smithi044 ceph-mon[58191]: pgmap v127: 161 pgs: 161 active+clean; 604 KiB data, 194 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:23:00.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:59 smithi044 ceph-mon[58191]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-17T07:23:00.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:59 smithi044 ceph-mon[58191]: Upgrade: osd.4 is safe to restart 2024-09-17T07:23:00.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:59 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:00.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:59 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-17T07:23:00.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:22:59 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:23:00.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:59 smithi044 ceph-mon[60356]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:23:00.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:59 smithi044 ceph-mon[60356]: pgmap v126: 161 pgs: 161 active+clean; 604 KiB data, 194 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:23:00.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:59 smithi044 ceph-mon[60356]: pgmap v127: 161 pgs: 161 active+clean; 604 KiB data, 194 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:23:00.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:59 smithi044 ceph-mon[60356]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-17T07:23:00.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:59 smithi044 ceph-mon[60356]: Upgrade: osd.4 is safe to restart 2024-09-17T07:23:00.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:59 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:00.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:59 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-17T07:23:00.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:22:59 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:23:00.950 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:00 smithi179 ceph-mon[50085]: Upgrade: Updating osd.4 2024-09-17T07:23:00.950 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:00 smithi179 ceph-mon[50085]: Deploying daemon osd.4 on smithi179 2024-09-17T07:23:01.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:00 smithi044 ceph-mon[58191]: Upgrade: Updating osd.4 2024-09-17T07:23:01.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:00 smithi044 ceph-mon[58191]: Deploying daemon osd.4 on smithi179 2024-09-17T07:23:01.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:00 smithi044 ceph-mon[60356]: Upgrade: Updating osd.4 2024-09-17T07:23:01.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:00 smithi044 ceph-mon[60356]: Deploying daemon osd.4 on smithi179 2024-09-17T07:23:01.848 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:01 smithi179 ceph-mon[50085]: pgmap v128: 161 pgs: 161 active+clean; 604 KiB data, 194 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:23:02.037 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:01 smithi044 ceph-mon[58191]: pgmap v128: 161 pgs: 161 active+clean; 604 KiB data, 194 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:23:02.037 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:01 smithi044 ceph-mon[60356]: pgmap v128: 161 pgs: 161 active+clean; 604 KiB data, 194 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:23:02.864 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:02 smithi179 systemd[1]: Stopping Ceph osd.4 for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:23:02.864 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.4[26649]: debug 2024-09-17T07:23:02.660+0000 7fa9623d0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-17T07:23:02.864 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.4[26649]: debug 2024-09-17T07:23:02.660+0000 7fa9623d0700 -1 osd.4 96 *** Got signal Terminated *** 2024-09-17T07:23:02.864 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.4[26649]: debug 2024-09-17T07:23:02.660+0000 7fa9623d0700 -1 osd.4 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T07:23:02.864 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:02 smithi179 podman[53233]: 2024-09-17 07:23:02.702181201 +0000 UTC m=+0.166039863 container died 911e6f47c822db206f0205541d7219f04b752cc2ca0a968ae2576e7e3eb47cb5 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.4, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20201204, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, maintainer=Dimitri Savineau , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image, CEPH_POINT_RELEASE=-16.2.0, GIT_CLEAN=True, RELEASE=HEAD) 2024-09-17T07:23:03.473 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:03 smithi179 podman[53233]: 2024-09-17 07:23:03.21964874 +0000 UTC m=+0.683507400 container cleanup 911e6f47c822db206f0205541d7219f04b752cc2ca0a968ae2576e7e3eb47cb5 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.4, CEPH_POINT_RELEASE=-16.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, maintainer=Dimitri Savineau , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image, RELEASE=HEAD, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20201204) 2024-09-17T07:23:03.474 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:03 smithi179 bash[53233]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.4 2024-09-17T07:23:03.474 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:03 smithi179 podman[53244]: 2024-09-17 07:23:03.438236162 +0000 UTC m=+0.733320560 container remove 911e6f47c822db206f0205541d7219f04b752cc2ca0a968ae2576e7e3eb47cb5 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.4, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-16.2.0, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20201204, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, org.label-schema.name=CentOS Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Dimitri Savineau ) 2024-09-17T07:23:03.972 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:03 smithi179 podman[53357]: 2024-09-17 07:23:03.843933915 +0000 UTC m=+0.016061115 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:23:03.973 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:03 smithi179 ceph-mon[50085]: pgmap v129: 161 pgs: 161 active+clean; 604 KiB data, 194 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:23:03.973 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:03 smithi179 ceph-mon[50085]: osd.4 reported immediately failed by osd.6 2024-09-17T07:23:03.973 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:03 smithi179 ceph-mon[50085]: osd.4 failed (root=default,host=smithi179) (connection refused reported by osd.6) 2024-09-17T07:23:03.973 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:03 smithi179 ceph-mon[50085]: osd.4 reported immediately failed by osd.0 2024-09-17T07:23:03.973 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:03 smithi179 ceph-mon[50085]: osd.4 reported immediately failed by osd.1 2024-09-17T07:23:03.973 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:03 smithi179 ceph-mon[50085]: osd.4 reported immediately failed by osd.2 2024-09-17T07:23:03.973 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:03 smithi179 ceph-mon[50085]: osd.4 reported immediately failed by osd.7 2024-09-17T07:23:03.973 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:03 smithi179 ceph-mon[50085]: osd.4 reported immediately failed by osd.6 2024-09-17T07:23:03.973 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:03 smithi179 ceph-mon[50085]: osd.4 reported immediately failed by osd.5 2024-09-17T07:23:03.973 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:03 smithi179 ceph-mon[50085]: osd.4 reported immediately failed by osd.0 2024-09-17T07:23:03.973 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:03 smithi179 ceph-mon[50085]: osd.4 reported immediately failed by osd.1 2024-09-17T07:23:03.973 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:03 smithi179 ceph-mon[50085]: osd.4 reported immediately failed by osd.2 2024-09-17T07:23:03.974 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:03 smithi179 ceph-mon[50085]: osd.4 reported immediately failed by osd.7 2024-09-17T07:23:03.974 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:03 smithi179 ceph-mon[50085]: osd.4 reported immediately failed by osd.5 2024-09-17T07:23:03.974 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:03 smithi179 ceph-mon[50085]: osd.4 reported immediately failed by osd.3 2024-09-17T07:23:03.974 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:03 smithi179 ceph-mon[50085]: osd.4 reported immediately failed by osd.3 2024-09-17T07:23:03.974 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:03 smithi179 ceph-mon[50085]: osd.4 reported immediately failed by osd.6 2024-09-17T07:23:03.974 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:03 smithi179 ceph-mon[50085]: osd.4 reported immediately failed by osd.2 2024-09-17T07:23:03.974 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:03 smithi179 ceph-mon[50085]: osd.4 reported immediately failed by osd.6 2024-09-17T07:23:03.974 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:03 smithi179 ceph-mon[50085]: osd.4 reported immediately failed by osd.5 2024-09-17T07:23:03.974 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:03 smithi179 ceph-mon[50085]: osd.4 reported immediately failed by osd.0 2024-09-17T07:23:03.974 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:03 smithi179 ceph-mon[50085]: osd.4 reported immediately failed by osd.1 2024-09-17T07:23:03.974 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:03 smithi179 ceph-mon[50085]: osd.4 reported immediately failed by osd.2 2024-09-17T07:23:03.974 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:03 smithi179 ceph-mon[50085]: osd.4 reported immediately failed by osd.7 2024-09-17T07:23:03.974 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:03 smithi179 ceph-mon[50085]: osd.4 reported immediately failed by osd.6 2024-09-17T07:23:03.975 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:03 smithi179 ceph-mon[50085]: osd.4 reported immediately failed by osd.5 2024-09-17T07:23:03.975 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:03 smithi179 ceph-mon[50085]: osd.4 reported immediately failed by osd.0 2024-09-17T07:23:03.975 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:03 smithi179 ceph-mon[50085]: osd.4 reported immediately failed by osd.1 2024-09-17T07:23:03.975 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:03 smithi179 ceph-mon[50085]: osd.4 reported immediately failed by osd.7 2024-09-17T07:23:03.975 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:03 smithi179 ceph-mon[50085]: osd.4 reported immediately failed by osd.3 2024-09-17T07:23:03.975 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:03 smithi179 ceph-mon[50085]: osd.4 reported immediately failed by osd.3 2024-09-17T07:23:03.975 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:03 smithi179 ceph-mon[50085]: from='client.44724 172.21.15.44:0/706417151' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:23:03.975 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:03 smithi179 ceph-mon[50085]: from='client.54613 172.21.15.44:0/1050344926' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:23:04.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[60356]: pgmap v129: 161 pgs: 161 active+clean; 604 KiB data, 194 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:23:04.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[60356]: osd.4 reported immediately failed by osd.6 2024-09-17T07:23:04.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[60356]: osd.4 failed (root=default,host=smithi179) (connection refused reported by osd.6) 2024-09-17T07:23:04.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[60356]: osd.4 reported immediately failed by osd.0 2024-09-17T07:23:04.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[60356]: osd.4 reported immediately failed by osd.1 2024-09-17T07:23:04.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[60356]: osd.4 reported immediately failed by osd.2 2024-09-17T07:23:04.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[60356]: osd.4 reported immediately failed by osd.7 2024-09-17T07:23:04.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[60356]: osd.4 reported immediately failed by osd.6 2024-09-17T07:23:04.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[60356]: osd.4 reported immediately failed by osd.5 2024-09-17T07:23:04.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[60356]: osd.4 reported immediately failed by osd.0 2024-09-17T07:23:04.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[60356]: osd.4 reported immediately failed by osd.1 2024-09-17T07:23:04.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[60356]: osd.4 reported immediately failed by osd.2 2024-09-17T07:23:04.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[60356]: osd.4 reported immediately failed by osd.7 2024-09-17T07:23:04.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[60356]: osd.4 reported immediately failed by osd.5 2024-09-17T07:23:04.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[60356]: osd.4 reported immediately failed by osd.3 2024-09-17T07:23:04.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[60356]: osd.4 reported immediately failed by osd.3 2024-09-17T07:23:04.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[60356]: osd.4 reported immediately failed by osd.6 2024-09-17T07:23:04.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[60356]: osd.4 reported immediately failed by osd.2 2024-09-17T07:23:04.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[60356]: osd.4 reported immediately failed by osd.6 2024-09-17T07:23:04.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[60356]: osd.4 reported immediately failed by osd.5 2024-09-17T07:23:04.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[60356]: osd.4 reported immediately failed by osd.0 2024-09-17T07:23:04.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[60356]: osd.4 reported immediately failed by osd.1 2024-09-17T07:23:04.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[60356]: osd.4 reported immediately failed by osd.2 2024-09-17T07:23:04.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[60356]: osd.4 reported immediately failed by osd.7 2024-09-17T07:23:04.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[60356]: osd.4 reported immediately failed by osd.6 2024-09-17T07:23:04.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[60356]: osd.4 reported immediately failed by osd.5 2024-09-17T07:23:04.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[60356]: osd.4 reported immediately failed by osd.0 2024-09-17T07:23:04.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[60356]: osd.4 reported immediately failed by osd.1 2024-09-17T07:23:04.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[60356]: osd.4 reported immediately failed by osd.7 2024-09-17T07:23:04.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[60356]: osd.4 reported immediately failed by osd.3 2024-09-17T07:23:04.078 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[60356]: osd.4 reported immediately failed by osd.3 2024-09-17T07:23:04.078 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[60356]: from='client.44724 172.21.15.44:0/706417151' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:23:04.078 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[60356]: from='client.54613 172.21.15.44:0/1050344926' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:23:04.078 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[58191]: pgmap v129: 161 pgs: 161 active+clean; 604 KiB data, 194 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:23:04.078 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[58191]: osd.4 reported immediately failed by osd.6 2024-09-17T07:23:04.078 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[58191]: osd.4 failed (root=default,host=smithi179) (connection refused reported by osd.6) 2024-09-17T07:23:04.078 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[58191]: osd.4 reported immediately failed by osd.0 2024-09-17T07:23:04.078 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[58191]: osd.4 reported immediately failed by osd.1 2024-09-17T07:23:04.078 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[58191]: osd.4 reported immediately failed by osd.2 2024-09-17T07:23:04.078 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[58191]: osd.4 reported immediately failed by osd.7 2024-09-17T07:23:04.078 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[58191]: osd.4 reported immediately failed by osd.6 2024-09-17T07:23:04.079 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[58191]: osd.4 reported immediately failed by osd.5 2024-09-17T07:23:04.079 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[58191]: osd.4 reported immediately failed by osd.0 2024-09-17T07:23:04.079 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[58191]: osd.4 reported immediately failed by osd.1 2024-09-17T07:23:04.079 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[58191]: osd.4 reported immediately failed by osd.2 2024-09-17T07:23:04.079 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[58191]: osd.4 reported immediately failed by osd.7 2024-09-17T07:23:04.079 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[58191]: osd.4 reported immediately failed by osd.5 2024-09-17T07:23:04.079 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[58191]: osd.4 reported immediately failed by osd.3 2024-09-17T07:23:04.079 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[58191]: osd.4 reported immediately failed by osd.3 2024-09-17T07:23:04.079 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[58191]: osd.4 reported immediately failed by osd.6 2024-09-17T07:23:04.079 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[58191]: osd.4 reported immediately failed by osd.2 2024-09-17T07:23:04.079 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[58191]: osd.4 reported immediately failed by osd.6 2024-09-17T07:23:04.079 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[58191]: osd.4 reported immediately failed by osd.5 2024-09-17T07:23:04.079 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[58191]: osd.4 reported immediately failed by osd.0 2024-09-17T07:23:04.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[58191]: osd.4 reported immediately failed by osd.1 2024-09-17T07:23:04.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[58191]: osd.4 reported immediately failed by osd.2 2024-09-17T07:23:04.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[58191]: osd.4 reported immediately failed by osd.7 2024-09-17T07:23:04.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[58191]: osd.4 reported immediately failed by osd.6 2024-09-17T07:23:04.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[58191]: osd.4 reported immediately failed by osd.5 2024-09-17T07:23:04.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[58191]: osd.4 reported immediately failed by osd.0 2024-09-17T07:23:04.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[58191]: osd.4 reported immediately failed by osd.1 2024-09-17T07:23:04.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[58191]: osd.4 reported immediately failed by osd.7 2024-09-17T07:23:04.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[58191]: osd.4 reported immediately failed by osd.3 2024-09-17T07:23:04.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[58191]: osd.4 reported immediately failed by osd.3 2024-09-17T07:23:04.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[58191]: from='client.44724 172.21.15.44:0/706417151' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:23:04.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:03 smithi044 ceph-mon[58191]: from='client.54613 172.21.15.44:0/1050344926' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:23:04.300 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:03 smithi179 podman[53357]: 2024-09-17 07:23:03.972714013 +0000 UTC m=+0.144841198 container create 8b2f77a009186d9d7ada55757e6bdde3aa5292c805889fe94aa0bee561d99ecd (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-17T07:23:04.300 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:04 smithi179 podman[53357]: 2024-09-17 07:23:04.208384089 +0000 UTC m=+0.380511277 container init 8b2f77a009186d9d7ada55757e6bdde3aa5292c805889fe94aa0bee561d99ecd (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4-deactivate, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T07:23:04.300 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:04 smithi179 podman[53357]: 2024-09-17 07:23:04.211727393 +0000 UTC m=+0.383854579 container start 8b2f77a009186d9d7ada55757e6bdde3aa5292c805889fe94aa0bee561d99ecd (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-17T07:23:04.300 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:04 smithi179 podman[53357]: 2024-09-17 07:23:04.300084572 +0000 UTC m=+0.472211764 container attach 8b2f77a009186d9d7ada55757e6bdde3aa5292c805889fe94aa0bee561d99ecd (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-17T07:23:04.427 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-09-17T07:23:04.614 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:04 smithi179 podman[53357]: 2024-09-17 07:23:04.396108369 +0000 UTC m=+0.568235557 container died 8b2f77a009186d9d7ada55757e6bdde3aa5292c805889fe94aa0bee561d99ecd (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, ceph=True, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T07:23:05.011 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:04 smithi044 ceph-mon[58191]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T07:23:05.011 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:04 smithi044 ceph-mon[58191]: osdmap e97: 8 total, 7 up, 8 in 2024-09-17T07:23:05.012 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:04 smithi044 ceph-mon[60356]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T07:23:05.012 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:04 smithi044 ceph-mon[60356]: osdmap e97: 8 total, 7 up, 8 in 2024-09-17T07:23:05.028 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:04 smithi179 ceph-mon[50085]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T07:23:05.028 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:04 smithi179 ceph-mon[50085]: osdmap e97: 8 total, 7 up, 8 in 2024-09-17T07:23:05.126 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T07:23:05.126 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.a smithi044 *:9093,9094 running (4m) 11s ago 9m 18.9M - 0.20.0 0881eb8f169f 0473d4b023ad 2024-09-17T07:23:05.126 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.a smithi179 *:3000 running (4m) 2m ago 9m 27.2M - 6.7.4 557c83e11646 43ab0e03dcac 2024-09-17T07:23:05.126 INFO:teuthology.orchestra.run.smithi044.stdout:iscsi.foo.smithi044.elmtfk smithi044 unknown 11s ago 8m - - 2024-09-17T07:23:05.126 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.x smithi179 *:8443,8765 running (3m) 2m ago 13m 436M - 18.2.4-855-g19e2a2be 980a0bfcd684 66b52dc26022 2024-09-17T07:23:05.126 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.y smithi044 *:8443,8765 running (4m) 11s ago 14m 482M - 18.2.4-855-g19e2a2be 980a0bfcd684 8080dd318a1c 2024-09-17T07:23:05.126 INFO:teuthology.orchestra.run.smithi044.stdout:mon.a smithi044 running (3m) 11s ago 15m 48.7M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 c3293e8cdbba 2024-09-17T07:23:05.126 INFO:teuthology.orchestra.run.smithi044.stdout:mon.b smithi179 running (2m) 2m ago 13m 27.7M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 ade749759184 2024-09-17T07:23:05.126 INFO:teuthology.orchestra.run.smithi044.stdout:mon.c smithi044 running (2m) 11s ago 13m 42.0M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 02f85d31e0bd 2024-09-17T07:23:05.126 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.a smithi044 *:9100 running (4m) 11s ago 9m 17.1M - 0.18.1 e5a616e4b9cf 8a2cde5bd9ed 2024-09-17T07:23:05.126 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.b smithi179 *:9100 running (4m) 2m ago 10m 8929k - 0.18.1 e5a616e4b9cf 04c63b5cb524 2024-09-17T07:23:05.127 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (109s) 11s ago 13m 52.4M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 3ba47a4e2e95 2024-09-17T07:23:05.127 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (76s) 11s ago 12m 50.5M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 7655e43d9944 2024-09-17T07:23:05.127 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (38s) 11s ago 12m 49.0M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 43a48289eada 2024-09-17T07:23:05.127 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi044 running (18s) 11s ago 12m 11.5M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 b4323c928f39 2024-09-17T07:23:05.127 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi179 running (11m) 2m ago 11m 51.7M 4096M 16.2.0 24ecd6d5f14c 911e6f47c822 2024-09-17T07:23:05.127 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi179 running (11m) 2m ago 11m 47.8M 4096M 16.2.0 24ecd6d5f14c e56060742dff 2024-09-17T07:23:05.127 INFO:teuthology.orchestra.run.smithi044.stdout:osd.6 smithi179 running (11m) 2m ago 11m 47.4M 4096M 16.2.0 24ecd6d5f14c 7a5c30b68cbc 2024-09-17T07:23:05.127 INFO:teuthology.orchestra.run.smithi044.stdout:osd.7 smithi179 running (11m) 2m ago 11m 53.5M 4096M 16.2.0 24ecd6d5f14c b0d186fbf8a6 2024-09-17T07:23:05.127 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.a smithi179 *:9095 unknown 2m ago 10m - - 2024-09-17T07:23:05.127 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi044.nhyhnh smithi044 *:8000 running (8m) 11s ago 8m 39.1M - 16.2.0 24ecd6d5f14c 9b455479614f 2024-09-17T07:23:05.127 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi179.tqlzyy smithi179 *:8000 running (8m) 2m ago 8m 41.2M - 16.2.0 24ecd6d5f14c c8d1693241ce 2024-09-17T07:23:05.127 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi044.vnufiz smithi044 *:80 running (8m) 11s ago 8m 39.2M - 16.2.0 24ecd6d5f14c 6eae2bff5ec8 2024-09-17T07:23:05.127 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi179.ecjhll smithi179 *:80 running (8m) 2m ago 8m 41.7M - 16.2.0 24ecd6d5f14c 253109ee0394 2024-09-17T07:23:05.364 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:05 smithi179 podman[53380]: 2024-09-17 07:23:05.02859078 +0000 UTC m=+0.626330609 container remove 8b2f77a009186d9d7ada55757e6bdde3aa5292c805889fe94aa0bee561d99ecd (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T07:23:05.364 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:05 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.4.service: Deactivated successfully. 2024-09-17T07:23:05.364 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:05 smithi179 systemd[1]: Stopped Ceph osd.4 for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:23:05.364 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:05 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.4.service: Consumed 1.159s CPU time. 2024-09-17T07:23:05.364 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:05 smithi179 systemd[1]: Starting Ceph osd.4 for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:23:05.529 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:23:05.529 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-09-17T07:23:05.529 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3 2024-09-17T07:23:05.529 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:23:05.529 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-09-17T07:23:05.529 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T07:23:05.529 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:23:05.529 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-09-17T07:23:05.529 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 3, 2024-09-17T07:23:05.529 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 4 2024-09-17T07:23:05.529 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:23:05.530 INFO:teuthology.orchestra.run.smithi044.stdout: "rgw": { 2024-09-17T07:23:05.530 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2024-09-17T07:23:05.530 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:23:05.530 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-09-17T07:23:05.530 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 7, 2024-09-17T07:23:05.530 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 9 2024-09-17T07:23:05.530 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-09-17T07:23:05.530 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:23:05.727 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:05 smithi179 podman[53478]: 2024-09-17 07:23:05.378559398 +0000 UTC m=+0.016458468 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:23:05.727 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:05 smithi179 podman[53478]: 2024-09-17 07:23:05.513185153 +0000 UTC m=+0.151084212 container create 42e8622bc418658c9176417f9f1919ede935c376518c7cbf6383de6530d41faa (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4-activate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-17T07:23:05.727 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:05 smithi179 podman[53478]: 2024-09-17 07:23:05.723675895 +0000 UTC m=+0.361574968 container init 42e8622bc418658c9176417f9f1919ede935c376518c7cbf6383de6530d41faa (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-17T07:23:05.727 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:05 smithi179 podman[53478]: 2024-09-17 07:23:05.727206528 +0000 UTC m=+0.365105590 container start 42e8622bc418658c9176417f9f1919ede935c376518c7cbf6383de6530d41faa (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4-activate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True) 2024-09-17T07:23:05.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:05 smithi044 ceph-mon[58191]: pgmap v131: 161 pgs: 23 stale+active+clean, 138 active+clean; 604 KiB data, 194 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:23:05.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:05 smithi044 ceph-mon[58191]: from='client.44736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:23:05.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:05 smithi044 ceph-mon[58191]: from='client.44742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:23:05.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:05 smithi044 ceph-mon[58191]: osdmap e98: 8 total, 7 up, 8 in 2024-09-17T07:23:05.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:05 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:05.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:05 smithi044 ceph-mon[58191]: from='client.44754 172.21.15.44:0/2212653964' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:23:05.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:05 smithi044 ceph-mon[60356]: pgmap v131: 161 pgs: 23 stale+active+clean, 138 active+clean; 604 KiB data, 194 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:23:05.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:05 smithi044 ceph-mon[60356]: from='client.44736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:23:05.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:05 smithi044 ceph-mon[60356]: from='client.44742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:23:05.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:05 smithi044 ceph-mon[60356]: osdmap e98: 8 total, 7 up, 8 in 2024-09-17T07:23:05.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:05 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:05.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:05 smithi044 ceph-mon[60356]: from='client.44754 172.21.15.44:0/2212653964' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:23:05.868 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:23:05.868 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9", 2024-09-17T07:23:05.868 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-09-17T07:23:05.868 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T07:23:05.868 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [ 2024-09-17T07:23:05.868 INFO:teuthology.orchestra.run.smithi044.stdout: "mon", 2024-09-17T07:23:05.868 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr" 2024-09-17T07:23:05.868 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-09-17T07:23:05.868 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "9/23 daemons upgraded", 2024-09-17T07:23:05.869 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "Currently upgrading osd daemons", 2024-09-17T07:23:05.869 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-09-17T07:23:05.869 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:23:06.019 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:05 smithi179 podman[53478]: 2024-09-17 07:23:05.82403336 +0000 UTC m=+0.461932430 container attach 42e8622bc418658c9176417f9f1919ede935c376518c7cbf6383de6530d41faa (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-17T07:23:06.020 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:05 smithi179 ceph-mon[50085]: pgmap v131: 161 pgs: 23 stale+active+clean, 138 active+clean; 604 KiB data, 194 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:23:06.020 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:05 smithi179 ceph-mon[50085]: from='client.44736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:23:06.020 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:05 smithi179 ceph-mon[50085]: from='client.44742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:23:06.020 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:05 smithi179 ceph-mon[50085]: osdmap e98: 8 total, 7 up, 8 in 2024-09-17T07:23:06.020 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:05 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:06.020 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:05 smithi179 ceph-mon[50085]: from='client.44754 172.21.15.44:0/2212653964' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:23:06.269 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_WARN 2 failed cephadm daemon(s); 1 osds down 2024-09-17T07:23:06.269 INFO:teuthology.orchestra.run.smithi044.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T07:23:06.269 INFO:teuthology.orchestra.run.smithi044.stdout: daemon iscsi.foo.smithi044.elmtfk on smithi044 is in unknown state 2024-09-17T07:23:06.269 INFO:teuthology.orchestra.run.smithi044.stdout: daemon prometheus.a on smithi179 is in unknown state 2024-09-17T07:23:06.269 INFO:teuthology.orchestra.run.smithi044.stdout:[WRN] OSD_DOWN: 1 osds down 2024-09-17T07:23:06.269 INFO:teuthology.orchestra.run.smithi044.stdout: osd.4 (root=default,host=smithi179) is down 2024-09-17T07:23:06.364 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4-activate[53493]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-17T07:23:06.364 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:06 smithi179 bash[53478]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-17T07:23:06.365 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4-activate[53493]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-17T07:23:06.365 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:06 smithi179 bash[53478]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-17T07:23:06.365 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4-activate[53493]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-17T07:23:06.365 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:06 smithi179 bash[53478]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-17T07:23:06.365 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4-activate[53493]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-17T07:23:06.365 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:06 smithi179 bash[53478]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-17T07:23:06.365 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4-activate[53493]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-09-17T07:23:06.365 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:06 smithi179 bash[53478]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-09-17T07:23:06.365 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4-activate[53493]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-17T07:23:06.365 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:06 smithi179 bash[53478]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-17T07:23:06.365 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4-activate[53493]: --> ceph-volume raw activate successful for osd ID: 4 2024-09-17T07:23:06.365 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:06 smithi179 bash[53478]: --> ceph-volume raw activate successful for osd ID: 4 2024-09-17T07:23:06.365 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:06 smithi179 podman[53478]: 2024-09-17 07:23:06.067164833 +0000 UTC m=+0.705063901 container died 42e8622bc418658c9176417f9f1919ede935c376518c7cbf6383de6530d41faa (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4-activate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T07:23:07.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:06 smithi044 ceph-mon[58191]: from='client.44748 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:23:07.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:06 smithi044 ceph-mon[58191]: from='client.54643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:23:07.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:06 smithi044 ceph-mon[58191]: from='client.54649 172.21.15.44:0/3997725312' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:23:07.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:06 smithi044 ceph-mon[60356]: from='client.44748 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:23:07.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:06 smithi044 ceph-mon[60356]: from='client.54643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:23:07.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:06 smithi044 ceph-mon[60356]: from='client.54649 172.21.15.44:0/3997725312' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:23:07.088 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:06 smithi179 podman[53611]: 2024-09-17 07:23:06.784017054 +0000 UTC m=+0.711113273 container remove 42e8622bc418658c9176417f9f1919ede935c376518c7cbf6383de6530d41faa (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4-activate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-17T07:23:07.088 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:06 smithi179 podman[53673]: 2024-09-17 07:23:06.900200314 +0000 UTC m=+0.016458531 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:23:07.088 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:07 smithi179 podman[53673]: 2024-09-17 07:23:07.003432757 +0000 UTC m=+0.119691019 container create 917adc8feb0bbbd3f9514ac1b01c37bba11ecb0d72d70fa6e7e99db8977404d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-17T07:23:07.089 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:06 smithi179 ceph-mon[50085]: from='client.44748 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:23:07.089 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:06 smithi179 ceph-mon[50085]: from='client.54643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:23:07.089 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:06 smithi179 ceph-mon[50085]: from='client.54649 172.21.15.44:0/3997725312' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:23:07.364 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:07 smithi179 podman[53673]: 2024-09-17 07:23:07.239326522 +0000 UTC m=+0.355584732 container init 917adc8feb0bbbd3f9514ac1b01c37bba11ecb0d72d70fa6e7e99db8977404d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-17T07:23:07.364 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:07 smithi179 podman[53673]: 2024-09-17 07:23:07.242439354 +0000 UTC m=+0.358697558 container start 917adc8feb0bbbd3f9514ac1b01c37bba11ecb0d72d70fa6e7e99db8977404d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-17T07:23:07.364 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:07 smithi179 ceph-osd[53691]: -- 172.21.15.179:0/2632897004 <== mon.2 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556f0e24f1e0 con 0x556f0e1edc00 2024-09-17T07:23:07.783 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:07 smithi179 bash[53673]: 917adc8feb0bbbd3f9514ac1b01c37bba11ecb0d72d70fa6e7e99db8977404d4 2024-09-17T07:23:07.783 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:07 smithi179 systemd[1]: Started Ceph osd.4 for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:23:08.073 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4[53687]: 2024-09-17T07:23:07.789+0000 7f1138680640 -1 Falling back to public interface 2024-09-17T07:23:08.073 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:07 smithi179 ceph-mon[50085]: pgmap v133: 161 pgs: 37 active+undersized, 3 stale+active+clean, 21 active+undersized+degraded, 100 active+clean; 604 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 142/999 objects degraded (14.214%) 2024-09-17T07:23:08.073 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:07 smithi179 ceph-mon[50085]: Health check failed: Degraded data redundancy: 142/999 objects degraded (14.214%), 21 pgs degraded (PG_DEGRADED) 2024-09-17T07:23:08.073 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:07 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:08.073 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:07 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:08.074 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:07 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:23:08.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:07 smithi044 ceph-mon[60356]: pgmap v133: 161 pgs: 37 active+undersized, 3 stale+active+clean, 21 active+undersized+degraded, 100 active+clean; 604 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 142/999 objects degraded (14.214%) 2024-09-17T07:23:08.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:07 smithi044 ceph-mon[60356]: Health check failed: Degraded data redundancy: 142/999 objects degraded (14.214%), 21 pgs degraded (PG_DEGRADED) 2024-09-17T07:23:08.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:07 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:08.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:07 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:08.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:07 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:23:08.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:07 smithi044 ceph-mon[58191]: pgmap v133: 161 pgs: 37 active+undersized, 3 stale+active+clean, 21 active+undersized+degraded, 100 active+clean; 604 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 142/999 objects degraded (14.214%) 2024-09-17T07:23:08.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:07 smithi044 ceph-mon[58191]: Health check failed: Degraded data redundancy: 142/999 objects degraded (14.214%), 21 pgs degraded (PG_DEGRADED) 2024-09-17T07:23:08.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:07 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:08.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:07 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:08.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:07 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:23:10.073 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:09 smithi179 ceph-mon[50085]: pgmap v134: 161 pgs: 37 active+undersized, 3 stale+active+clean, 21 active+undersized+degraded, 100 active+clean; 604 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 142/999 objects degraded (14.214%) 2024-09-17T07:23:10.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:09 smithi044 ceph-mon[60356]: pgmap v134: 161 pgs: 37 active+undersized, 3 stale+active+clean, 21 active+undersized+degraded, 100 active+clean; 604 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 142/999 objects degraded (14.214%) 2024-09-17T07:23:10.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:09 smithi044 ceph-mon[58191]: pgmap v134: 161 pgs: 37 active+undersized, 3 stale+active+clean, 21 active+undersized+degraded, 100 active+clean; 604 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 142/999 objects degraded (14.214%) 2024-09-17T07:23:10.787 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4[53687]: 2024-09-17T07:23:10.505+0000 7f1138680640 -1 osd.4 0 read_superblock omap replica is missing. 2024-09-17T07:23:10.787 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4[53687]: 2024-09-17T07:23:10.543+0000 7f1138680640 -1 osd.4 96 log_to_monitors true 2024-09-17T07:23:11.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:10 smithi044 ceph-mon[60356]: from='osd.4 [v2:172.21.15.179:6800/3130396509,v1:172.21.15.179:6801/3130396509]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T07:23:11.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:10 smithi044 ceph-mon[60356]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T07:23:11.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:10 smithi044 ceph-mon[58191]: from='osd.4 [v2:172.21.15.179:6800/3130396509,v1:172.21.15.179:6801/3130396509]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T07:23:11.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:10 smithi044 ceph-mon[58191]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T07:23:11.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:10 smithi179 ceph-mon[50085]: from='osd.4 [v2:172.21.15.179:6800/3130396509,v1:172.21.15.179:6801/3130396509]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T07:23:11.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:10 smithi179 ceph-mon[50085]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T07:23:11.114 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:23:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4[53687]: 2024-09-17T07:23:10.813+0000 7f113042a640 -1 osd.4 96 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T07:23:12.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:11 smithi044 ceph-mon[60356]: pgmap v135: 161 pgs: 43 active+undersized, 25 active+undersized+degraded, 93 active+clean; 604 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 164/999 objects degraded (16.416%) 2024-09-17T07:23:12.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:11 smithi044 ceph-mon[60356]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-17T07:23:12.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:11 smithi044 ceph-mon[60356]: osdmap e99: 8 total, 7 up, 8 in 2024-09-17T07:23:12.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:11 smithi044 ceph-mon[60356]: from='osd.4 [v2:172.21.15.179:6800/3130396509,v1:172.21.15.179:6801/3130396509]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:23:12.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:11 smithi044 ceph-mon[60356]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:23:12.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:11 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:12.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:11 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:12.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:11 smithi044 ceph-mon[58191]: pgmap v135: 161 pgs: 43 active+undersized, 25 active+undersized+degraded, 93 active+clean; 604 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 164/999 objects degraded (16.416%) 2024-09-17T07:23:12.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:11 smithi044 ceph-mon[58191]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-17T07:23:12.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:11 smithi044 ceph-mon[58191]: osdmap e99: 8 total, 7 up, 8 in 2024-09-17T07:23:12.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:11 smithi044 ceph-mon[58191]: from='osd.4 [v2:172.21.15.179:6800/3130396509,v1:172.21.15.179:6801/3130396509]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:23:12.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:11 smithi044 ceph-mon[58191]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:23:12.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:11 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:12.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:11 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:12.113 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:11 smithi179 ceph-mon[50085]: pgmap v135: 161 pgs: 43 active+undersized, 25 active+undersized+degraded, 93 active+clean; 604 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 164/999 objects degraded (16.416%) 2024-09-17T07:23:12.113 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:11 smithi179 ceph-mon[50085]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-17T07:23:12.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:11 smithi179 ceph-mon[50085]: osdmap e99: 8 total, 7 up, 8 in 2024-09-17T07:23:12.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:11 smithi179 ceph-mon[50085]: from='osd.4 [v2:172.21.15.179:6800/3130396509,v1:172.21.15.179:6801/3130396509]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:23:12.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:11 smithi179 ceph-mon[50085]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:23:12.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:11 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:12.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:11 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:12.915 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:12 smithi179 ceph-mon[50085]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T07:23:12.916 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:12 smithi179 ceph-mon[50085]: osd.4 [v2:172.21.15.179:6800/3130396509,v1:172.21.15.179:6801/3130396509] boot 2024-09-17T07:23:12.916 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:12 smithi179 ceph-mon[50085]: osdmap e100: 8 total, 8 up, 8 in 2024-09-17T07:23:12.916 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:12 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:23:12.916 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:12 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:12.916 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:12 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:12.916 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:12 smithi179 ceph-mon[50085]: osdmap e101: 8 total, 8 up, 8 in 2024-09-17T07:23:13.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:12 smithi044 ceph-mon[60356]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T07:23:13.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:12 smithi044 ceph-mon[60356]: osd.4 [v2:172.21.15.179:6800/3130396509,v1:172.21.15.179:6801/3130396509] boot 2024-09-17T07:23:13.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:12 smithi044 ceph-mon[60356]: osdmap e100: 8 total, 8 up, 8 in 2024-09-17T07:23:13.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:12 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:23:13.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:12 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:13.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:12 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:13.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:12 smithi044 ceph-mon[60356]: osdmap e101: 8 total, 8 up, 8 in 2024-09-17T07:23:13.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:12 smithi044 ceph-mon[58191]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T07:23:13.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:12 smithi044 ceph-mon[58191]: osd.4 [v2:172.21.15.179:6800/3130396509,v1:172.21.15.179:6801/3130396509] boot 2024-09-17T07:23:13.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:12 smithi044 ceph-mon[58191]: osdmap e100: 8 total, 8 up, 8 in 2024-09-17T07:23:13.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:12 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T07:23:13.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:12 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:13.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:12 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:13.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:12 smithi044 ceph-mon[58191]: osdmap e101: 8 total, 8 up, 8 in 2024-09-17T07:23:13.863 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:13 smithi179 ceph-mon[50085]: pgmap v138: 161 pgs: 43 active+undersized, 25 active+undersized+degraded, 93 active+clean; 604 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 164/999 objects degraded (16.416%) 2024-09-17T07:23:13.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:13 smithi179 ceph-mon[50085]: Health check update: Degraded data redundancy: 164/999 objects degraded (16.416%), 25 pgs degraded (PG_DEGRADED) 2024-09-17T07:23:14.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:13 smithi044 ceph-mon[60356]: pgmap v138: 161 pgs: 43 active+undersized, 25 active+undersized+degraded, 93 active+clean; 604 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 164/999 objects degraded (16.416%) 2024-09-17T07:23:14.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:13 smithi044 ceph-mon[60356]: Health check update: Degraded data redundancy: 164/999 objects degraded (16.416%), 25 pgs degraded (PG_DEGRADED) 2024-09-17T07:23:14.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:13 smithi044 ceph-mon[58191]: pgmap v138: 161 pgs: 43 active+undersized, 25 active+undersized+degraded, 93 active+clean; 604 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 164/999 objects degraded (16.416%) 2024-09-17T07:23:14.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:13 smithi044 ceph-mon[58191]: Health check update: Degraded data redundancy: 164/999 objects degraded (16.416%), 25 pgs degraded (PG_DEGRADED) 2024-09-17T07:23:15.679 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:23:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:23:15.595+0000 7f5a7ae47640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-09-17T07:23:15.995 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:15 smithi044 ceph-mon[58191]: pgmap v140: 161 pgs: 38 active+undersized, 23 active+undersized+degraded, 100 active+clean; 604 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 151/999 objects degraded (15.115%) 2024-09-17T07:23:15.995 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:15 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:15.995 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:15 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:15.995 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:15 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:23:15.995 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:15 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:23:15.995 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:15 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:15.995 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:15 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:23:15.995 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:15 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:23:15.995 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:15 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:23:15.995 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:15 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:23:15.995 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:15 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T07:23:15.996 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:15 smithi044 ceph-mon[60356]: pgmap v140: 161 pgs: 38 active+undersized, 23 active+undersized+degraded, 100 active+clean; 604 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 151/999 objects degraded (15.115%) 2024-09-17T07:23:15.996 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:15 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:15.996 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:15 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:15.996 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:15 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:23:15.996 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:15 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:23:15.996 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:15 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:15.996 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:15 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:23:15.996 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:15 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:23:15.996 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:15 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:23:15.996 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:15 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:23:15.996 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:15 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T07:23:16.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:15 smithi179 ceph-mon[50085]: pgmap v140: 161 pgs: 38 active+undersized, 23 active+undersized+degraded, 100 active+clean; 604 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 151/999 objects degraded (15.115%) 2024-09-17T07:23:16.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:15 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:16.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:15 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:16.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:15 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:23:16.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:15 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:23:16.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:15 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:16.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:15 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:23:16.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:15 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:23:16.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:15 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:23:16.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:15 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:23:16.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:15 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T07:23:17.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:16 smithi044 ceph-mon[58191]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:23:17.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:16 smithi044 ceph-mon[58191]: pgmap v141: 161 pgs: 35 active+undersized, 19 active+undersized+degraded, 107 active+clean; 604 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 124/999 objects degraded (12.412%) 2024-09-17T07:23:17.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:16 smithi044 ceph-mon[58191]: pgmap v142: 161 pgs: 35 active+undersized, 19 active+undersized+degraded, 107 active+clean; 604 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 124/999 objects degraded (12.412%) 2024-09-17T07:23:17.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:16 smithi044 ceph-mon[58191]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T07:23:17.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:16 smithi044 ceph-mon[58191]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-09-17T07:23:17.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:16 smithi044 ceph-mon[60356]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:23:17.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:16 smithi044 ceph-mon[60356]: pgmap v141: 161 pgs: 35 active+undersized, 19 active+undersized+degraded, 107 active+clean; 604 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 124/999 objects degraded (12.412%) 2024-09-17T07:23:17.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:16 smithi044 ceph-mon[60356]: pgmap v142: 161 pgs: 35 active+undersized, 19 active+undersized+degraded, 107 active+clean; 604 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 124/999 objects degraded (12.412%) 2024-09-17T07:23:17.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:16 smithi044 ceph-mon[60356]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T07:23:17.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:16 smithi044 ceph-mon[60356]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-09-17T07:23:17.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:16 smithi179 ceph-mon[50085]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:23:17.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:16 smithi179 ceph-mon[50085]: pgmap v141: 161 pgs: 35 active+undersized, 19 active+undersized+degraded, 107 active+clean; 604 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 124/999 objects degraded (12.412%) 2024-09-17T07:23:17.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:16 smithi179 ceph-mon[50085]: pgmap v142: 161 pgs: 35 active+undersized, 19 active+undersized+degraded, 107 active+clean; 604 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 124/999 objects degraded (12.412%) 2024-09-17T07:23:17.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:16 smithi179 ceph-mon[50085]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T07:23:17.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:16 smithi179 ceph-mon[50085]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-09-17T07:23:18.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:17 smithi179 ceph-mon[50085]: from='client.44784 172.21.15.44:0/3696964522' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:23:18.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:17 smithi179 ceph-mon[50085]: from='client.44790 172.21.15.44:0/1453487008' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:23:18.312 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:17 smithi044 ceph-mon[58191]: from='client.44784 172.21.15.44:0/3696964522' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:23:18.312 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:17 smithi044 ceph-mon[58191]: from='client.44790 172.21.15.44:0/1453487008' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:23:18.312 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:17 smithi044 ceph-mon[60356]: from='client.44784 172.21.15.44:0/3696964522' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:23:18.312 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:17 smithi044 ceph-mon[60356]: from='client.44790 172.21.15.44:0/1453487008' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:23:19.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:18 smithi044 ceph-mon[58191]: pgmap v143: 161 pgs: 161 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:23:19.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:18 smithi044 ceph-mon[58191]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 124/999 objects degraded (12.412%), 19 pgs degraded) 2024-09-17T07:23:19.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:18 smithi044 ceph-mon[60356]: pgmap v143: 161 pgs: 161 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:23:19.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:18 smithi044 ceph-mon[60356]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 124/999 objects degraded (12.412%), 19 pgs degraded) 2024-09-17T07:23:19.113 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:18 smithi179 ceph-mon[50085]: pgmap v143: 161 pgs: 161 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:23:19.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:18 smithi179 ceph-mon[50085]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 124/999 objects degraded (12.412%), 19 pgs degraded) 2024-09-17T07:23:21.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:20 smithi179 ceph-mon[50085]: pgmap v144: 161 pgs: 161 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:23:21.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:20 smithi044 ceph-mon[58191]: pgmap v144: 161 pgs: 161 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:23:21.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:20 smithi044 ceph-mon[60356]: pgmap v144: 161 pgs: 161 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:23:23.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:22 smithi179 ceph-mon[50085]: pgmap v145: 161 pgs: 161 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 118 B/s rd, 0 op/s 2024-09-17T07:23:23.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:22 smithi044 ceph-mon[58191]: pgmap v145: 161 pgs: 161 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 118 B/s rd, 0 op/s 2024-09-17T07:23:23.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:22 smithi044 ceph-mon[60356]: pgmap v145: 161 pgs: 161 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 118 B/s rd, 0 op/s 2024-09-17T07:23:25.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:24 smithi179 ceph-mon[50085]: pgmap v146: 161 pgs: 161 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 108 B/s rd, 0 op/s 2024-09-17T07:23:25.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:24 smithi044 ceph-mon[58191]: pgmap v146: 161 pgs: 161 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 108 B/s rd, 0 op/s 2024-09-17T07:23:25.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:24 smithi044 ceph-mon[60356]: pgmap v146: 161 pgs: 161 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 108 B/s rd, 0 op/s 2024-09-17T07:23:26.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:26 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:26.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:26 smithi044 ceph-mon[58191]: pgmap v147: 161 pgs: 161 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-09-17T07:23:26.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:26 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:26.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:26 smithi044 ceph-mon[60356]: pgmap v147: 161 pgs: 161 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-09-17T07:23:26.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:26 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:26.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:26 smithi179 ceph-mon[50085]: pgmap v147: 161 pgs: 161 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-09-17T07:23:28.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:28 smithi044 ceph-mon[58191]: pgmap v148: 161 pgs: 161 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:23:28.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:28 smithi044 ceph-mon[60356]: pgmap v148: 161 pgs: 161 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:23:28.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:28 smithi179 ceph-mon[50085]: pgmap v148: 161 pgs: 161 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:23:30.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:30 smithi044 ceph-mon[58191]: pgmap v149: 161 pgs: 161 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:23:30.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:30 smithi044 ceph-mon[60356]: pgmap v149: 161 pgs: 161 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:23:30.863 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:30 smithi179 ceph-mon[50085]: pgmap v149: 161 pgs: 161 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:23:31.540 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:31 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T07:23:31.540 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:31 smithi044 ceph-mon[60356]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T07:23:31.540 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:31 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T07:23:31.540 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:31 smithi044 ceph-mon[58191]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T07:23:31.540 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:31 smithi044 ceph-mon[58191]: Upgrade: osd.5 is safe to restart 2024-09-17T07:23:31.540 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:31 smithi044 ceph-mon[58191]: from='client.44808 172.21.15.44:0/2095586249' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:23:31.540 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:31 smithi044 ceph-mon[58191]: from='client.44814 172.21.15.44:0/349424944' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/1355368272"}]: access denied 2024-09-17T07:23:31.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:31 smithi044 ceph-mon[60356]: Upgrade: osd.5 is safe to restart 2024-09-17T07:23:31.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:31 smithi044 ceph-mon[60356]: from='client.44808 172.21.15.44:0/2095586249' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:23:31.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:31 smithi044 ceph-mon[60356]: from='client.44814 172.21.15.44:0/349424944' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/1355368272"}]: access denied 2024-09-17T07:23:31.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:31 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T07:23:31.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:31 smithi179 ceph-mon[50085]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T07:23:31.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:31 smithi179 ceph-mon[50085]: Upgrade: osd.5 is safe to restart 2024-09-17T07:23:31.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:31 smithi179 ceph-mon[50085]: from='client.44808 172.21.15.44:0/2095586249' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:23:31.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:31 smithi179 ceph-mon[50085]: from='client.44814 172.21.15.44:0/349424944' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/1355368272"}]: access denied 2024-09-17T07:23:32.563 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:32 smithi179 ceph-mon[50085]: pgmap v150: 161 pgs: 161 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-17T07:23:32.563 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:32 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:32.563 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:32 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-17T07:23:32.563 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:32 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:23:32.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:32 smithi044 ceph-mon[58191]: pgmap v150: 161 pgs: 161 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-17T07:23:32.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:32 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:32.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:32 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-17T07:23:32.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:32 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:23:32.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:32 smithi044 ceph-mon[60356]: pgmap v150: 161 pgs: 161 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-17T07:23:32.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:32 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:32.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:32 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-17T07:23:32.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:32 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:23:33.778 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:33 smithi179 ceph-mon[50085]: Upgrade: Updating osd.5 2024-09-17T07:23:33.778 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:33 smithi179 ceph-mon[50085]: Deploying daemon osd.5 on smithi179 2024-09-17T07:23:33.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:33 smithi044 ceph-mon[58191]: Upgrade: Updating osd.5 2024-09-17T07:23:33.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:33 smithi044 ceph-mon[58191]: Deploying daemon osd.5 on smithi179 2024-09-17T07:23:33.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:33 smithi044 ceph-mon[60356]: Upgrade: Updating osd.5 2024-09-17T07:23:33.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:33 smithi044 ceph-mon[60356]: Deploying daemon osd.5 on smithi179 2024-09-17T07:23:34.816 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:34 smithi179 ceph-mon[50085]: pgmap v151: 161 pgs: 161 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:23:34.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:34 smithi044 ceph-mon[58191]: pgmap v151: 161 pgs: 161 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:23:34.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:34 smithi044 ceph-mon[60356]: pgmap v151: 161 pgs: 161 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:23:35.114 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:34 smithi179 systemd[1]: Stopping Ceph osd.5 for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:23:35.114 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.5[29018]: debug 2024-09-17T07:23:34.985+0000 7f7e7850a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-17T07:23:35.114 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.5[29018]: debug 2024-09-17T07:23:34.985+0000 7f7e7850a700 -1 osd.5 101 *** Got signal Terminated *** 2024-09-17T07:23:35.114 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.5[29018]: debug 2024-09-17T07:23:34.985+0000 7f7e7850a700 -1 osd.5 101 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T07:23:35.114 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:35 smithi179 podman[55580]: 2024-09-17 07:23:35.006970177 +0000 UTC m=+0.146924145 container died e56060742dff3e4ea975742d44a69c6c30fd4af3b4aa7456f7b480da64bdb146 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Dimitri Savineau , org.label-schema.schema-version=1.0, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20201204, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Base Image, CEPH_POINT_RELEASE=-16.2.0, RELEASE=HEAD, ceph=True, org.label-schema.license=GPLv2) 2024-09-17T07:23:35.552 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:35 smithi179 podman[55580]: 2024-09-17 07:23:35.411524302 +0000 UTC m=+0.551478272 container cleanup e56060742dff3e4ea975742d44a69c6c30fd4af3b4aa7456f7b480da64bdb146 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image, CEPH_POINT_RELEASE=-16.2.0, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, org.label-schema.schema-version=1.0, maintainer=Dimitri Savineau , RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20201204, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True) 2024-09-17T07:23:35.552 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:35 smithi179 bash[55580]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.5 2024-09-17T07:23:35.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[58191]: osd.5 reported immediately failed by osd.2 2024-09-17T07:23:35.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[58191]: osd.5 failed (root=default,host=smithi179) (connection refused reported by osd.2) 2024-09-17T07:23:35.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[58191]: osd.5 reported immediately failed by osd.0 2024-09-17T07:23:35.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[58191]: osd.5 reported immediately failed by osd.4 2024-09-17T07:23:35.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[58191]: osd.5 reported immediately failed by osd.0 2024-09-17T07:23:35.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[58191]: osd.5 reported immediately failed by osd.0 2024-09-17T07:23:35.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[58191]: osd.5 reported immediately failed by osd.1 2024-09-17T07:23:35.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[58191]: osd.5 reported immediately failed by osd.2 2024-09-17T07:23:35.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[58191]: osd.5 reported immediately failed by osd.6 2024-09-17T07:23:35.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[58191]: osd.5 reported immediately failed by osd.0 2024-09-17T07:23:35.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[58191]: osd.5 reported immediately failed by osd.1 2024-09-17T07:23:35.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[58191]: osd.5 reported immediately failed by osd.2 2024-09-17T07:23:35.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[58191]: osd.5 reported immediately failed by osd.3 2024-09-17T07:23:35.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[58191]: osd.5 reported immediately failed by osd.6 2024-09-17T07:23:35.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[58191]: osd.5 reported immediately failed by osd.7 2024-09-17T07:23:35.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[58191]: osd.5 reported immediately failed by osd.3 2024-09-17T07:23:35.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[58191]: osd.5 reported immediately failed by osd.4 2024-09-17T07:23:35.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[58191]: osd.5 reported immediately failed by osd.7 2024-09-17T07:23:35.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[58191]: osd.5 reported immediately failed by osd.4 2024-09-17T07:23:35.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[60356]: osd.5 reported immediately failed by osd.2 2024-09-17T07:23:35.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[60356]: osd.5 failed (root=default,host=smithi179) (connection refused reported by osd.2) 2024-09-17T07:23:35.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[60356]: osd.5 reported immediately failed by osd.0 2024-09-17T07:23:35.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[60356]: osd.5 reported immediately failed by osd.4 2024-09-17T07:23:35.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[60356]: osd.5 reported immediately failed by osd.0 2024-09-17T07:23:35.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[60356]: osd.5 reported immediately failed by osd.0 2024-09-17T07:23:35.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[60356]: osd.5 reported immediately failed by osd.1 2024-09-17T07:23:35.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[60356]: osd.5 reported immediately failed by osd.2 2024-09-17T07:23:35.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[60356]: osd.5 reported immediately failed by osd.6 2024-09-17T07:23:35.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[60356]: osd.5 reported immediately failed by osd.0 2024-09-17T07:23:35.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[60356]: osd.5 reported immediately failed by osd.1 2024-09-17T07:23:35.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[60356]: osd.5 reported immediately failed by osd.2 2024-09-17T07:23:35.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[60356]: osd.5 reported immediately failed by osd.3 2024-09-17T07:23:35.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[60356]: osd.5 reported immediately failed by osd.6 2024-09-17T07:23:35.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[60356]: osd.5 reported immediately failed by osd.7 2024-09-17T07:23:35.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[60356]: osd.5 reported immediately failed by osd.3 2024-09-17T07:23:35.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[60356]: osd.5 reported immediately failed by osd.4 2024-09-17T07:23:35.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[60356]: osd.5 reported immediately failed by osd.7 2024-09-17T07:23:35.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:35 smithi044 ceph-mon[60356]: osd.5 reported immediately failed by osd.4 2024-09-17T07:23:35.863 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:35 smithi179 podman[55593]: 2024-09-17 07:23:35.654571945 +0000 UTC m=+0.641652780 container remove e56060742dff3e4ea975742d44a69c6c30fd4af3b4aa7456f7b480da64bdb146 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, RELEASE=HEAD, maintainer=Dimitri Savineau , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-16.2.0, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-09-17T07:23:35.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:35 smithi179 ceph-mon[50085]: osd.5 reported immediately failed by osd.2 2024-09-17T07:23:35.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:35 smithi179 ceph-mon[50085]: osd.5 failed (root=default,host=smithi179) (connection refused reported by osd.2) 2024-09-17T07:23:35.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:35 smithi179 ceph-mon[50085]: osd.5 reported immediately failed by osd.0 2024-09-17T07:23:35.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:35 smithi179 ceph-mon[50085]: osd.5 reported immediately failed by osd.4 2024-09-17T07:23:35.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:35 smithi179 ceph-mon[50085]: osd.5 reported immediately failed by osd.0 2024-09-17T07:23:35.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:35 smithi179 ceph-mon[50085]: osd.5 reported immediately failed by osd.0 2024-09-17T07:23:35.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:35 smithi179 ceph-mon[50085]: osd.5 reported immediately failed by osd.1 2024-09-17T07:23:35.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:35 smithi179 ceph-mon[50085]: osd.5 reported immediately failed by osd.2 2024-09-17T07:23:35.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:35 smithi179 ceph-mon[50085]: osd.5 reported immediately failed by osd.6 2024-09-17T07:23:35.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:35 smithi179 ceph-mon[50085]: osd.5 reported immediately failed by osd.0 2024-09-17T07:23:35.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:35 smithi179 ceph-mon[50085]: osd.5 reported immediately failed by osd.1 2024-09-17T07:23:35.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:35 smithi179 ceph-mon[50085]: osd.5 reported immediately failed by osd.2 2024-09-17T07:23:35.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:35 smithi179 ceph-mon[50085]: osd.5 reported immediately failed by osd.3 2024-09-17T07:23:35.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:35 smithi179 ceph-mon[50085]: osd.5 reported immediately failed by osd.6 2024-09-17T07:23:35.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:35 smithi179 ceph-mon[50085]: osd.5 reported immediately failed by osd.7 2024-09-17T07:23:35.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:35 smithi179 ceph-mon[50085]: osd.5 reported immediately failed by osd.3 2024-09-17T07:23:35.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:35 smithi179 ceph-mon[50085]: osd.5 reported immediately failed by osd.4 2024-09-17T07:23:35.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:35 smithi179 ceph-mon[50085]: osd.5 reported immediately failed by osd.7 2024-09-17T07:23:35.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:35 smithi179 ceph-mon[50085]: osd.5 reported immediately failed by osd.4 2024-09-17T07:23:36.364 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:36 smithi179 podman[55706]: 2024-09-17 07:23:36.040499101 +0000 UTC m=+0.016582414 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:23:36.364 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:36 smithi179 podman[55706]: 2024-09-17 07:23:36.147338536 +0000 UTC m=+0.123421877 container create b095455891799c184f3cdb9d8232caa8cd34d5cdf49919876dcd139847920165 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux ) 2024-09-17T07:23:36.651 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-09-17T07:23:36.657 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:36 smithi179 ceph-mon[50085]: pgmap v152: 161 pgs: 161 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:23:36.658 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:36 smithi179 ceph-mon[50085]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T07:23:36.658 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:36 smithi179 ceph-mon[50085]: osdmap e102: 8 total, 7 up, 8 in 2024-09-17T07:23:36.658 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:36 smithi179 podman[55706]: 2024-09-17 07:23:36.458134694 +0000 UTC m=+0.434217998 container init b095455891799c184f3cdb9d8232caa8cd34d5cdf49919876dcd139847920165 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-17T07:23:36.658 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:36 smithi179 podman[55706]: 2024-09-17 07:23:36.461449073 +0000 UTC m=+0.437532373 container start b095455891799c184f3cdb9d8232caa8cd34d5cdf49919876dcd139847920165 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD) 2024-09-17T07:23:36.658 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:36 smithi179 podman[55706]: 2024-09-17 07:23:36.658177413 +0000 UTC m=+0.634260715 container attach b095455891799c184f3cdb9d8232caa8cd34d5cdf49919876dcd139847920165 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T07:23:36.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:36 smithi044 ceph-mon[60356]: pgmap v152: 161 pgs: 161 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:23:36.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:36 smithi044 ceph-mon[60356]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T07:23:36.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:36 smithi044 ceph-mon[60356]: osdmap e102: 8 total, 7 up, 8 in 2024-09-17T07:23:36.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:36 smithi044 ceph-mon[58191]: pgmap v152: 161 pgs: 161 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T07:23:36.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:36 smithi044 ceph-mon[58191]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T07:23:36.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:36 smithi044 ceph-mon[58191]: osdmap e102: 8 total, 7 up, 8 in 2024-09-17T07:23:37.012 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:36 smithi179 podman[55706]: 2024-09-17 07:23:36.665265274 +0000 UTC m=+0.641348574 container died b095455891799c184f3cdb9d8232caa8cd34d5cdf49919876dcd139847920165 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5-deactivate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, ceph=True) 2024-09-17T07:23:37.316 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T07:23:37.316 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.a smithi044 *:9093,9094 running (5m) 43s ago 10m 18.9M - 0.20.0 0881eb8f169f 0473d4b023ad 2024-09-17T07:23:37.316 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.a smithi179 *:3000 running (4m) 25s ago 9m 33.2M - 6.7.4 557c83e11646 43ab0e03dcac 2024-09-17T07:23:37.316 INFO:teuthology.orchestra.run.smithi044.stdout:iscsi.foo.smithi044.elmtfk smithi044 unknown 43s ago 9m - - 2024-09-17T07:23:37.316 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.x smithi179 *:8443,8765 running (4m) 25s ago 14m 437M - 18.2.4-855-g19e2a2be 980a0bfcd684 66b52dc26022 2024-09-17T07:23:37.316 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.y smithi044 *:8443,8765 running (4m) 43s ago 15m 482M - 18.2.4-855-g19e2a2be 980a0bfcd684 8080dd318a1c 2024-09-17T07:23:37.316 INFO:teuthology.orchestra.run.smithi044.stdout:mon.a smithi044 running (3m) 43s ago 15m 48.7M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 c3293e8cdbba 2024-09-17T07:23:37.316 INFO:teuthology.orchestra.run.smithi044.stdout:mon.b smithi179 running (3m) 25s ago 14m 43.2M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 ade749759184 2024-09-17T07:23:37.316 INFO:teuthology.orchestra.run.smithi044.stdout:mon.c smithi044 running (3m) 43s ago 14m 42.0M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 02f85d31e0bd 2024-09-17T07:23:37.316 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.a smithi044 *:9100 running (5m) 43s ago 10m 17.1M - 0.18.1 e5a616e4b9cf 8a2cde5bd9ed 2024-09-17T07:23:37.316 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.b smithi179 *:9100 running (5m) 25s ago 10m 8929k - 0.18.1 e5a616e4b9cf 04c63b5cb524 2024-09-17T07:23:37.317 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (2m) 43s ago 13m 52.4M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 3ba47a4e2e95 2024-09-17T07:23:37.317 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (108s) 43s ago 13m 50.5M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 7655e43d9944 2024-09-17T07:23:37.317 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (70s) 43s ago 13m 49.0M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 43a48289eada 2024-09-17T07:23:37.317 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi044 running (50s) 43s ago 12m 11.5M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 b4323c928f39 2024-09-17T07:23:37.317 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi179 running (30s) 25s ago 12m 14.4M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 917adc8feb0b 2024-09-17T07:23:37.317 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi179 running (12m) 25s ago 12m 53.0M 4096M 16.2.0 24ecd6d5f14c e56060742dff 2024-09-17T07:23:37.317 INFO:teuthology.orchestra.run.smithi044.stdout:osd.6 smithi179 running (11m) 25s ago 11m 51.6M 4096M 16.2.0 24ecd6d5f14c 7a5c30b68cbc 2024-09-17T07:23:37.317 INFO:teuthology.orchestra.run.smithi044.stdout:osd.7 smithi179 running (11m) 25s ago 11m 57.5M 4096M 16.2.0 24ecd6d5f14c b0d186fbf8a6 2024-09-17T07:23:37.317 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.a smithi179 *:9095 error 25s ago 11m - - 2024-09-17T07:23:37.317 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi044.nhyhnh smithi044 *:8000 running (9m) 43s ago 9m 39.1M - 16.2.0 24ecd6d5f14c 9b455479614f 2024-09-17T07:23:37.317 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi179.tqlzyy smithi179 *:8000 running (9m) 25s ago 9m 41.8M - 16.2.0 24ecd6d5f14c c8d1693241ce 2024-09-17T07:23:37.317 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi044.vnufiz smithi044 *:80 running (9m) 43s ago 9m 39.2M - 16.2.0 24ecd6d5f14c 6eae2bff5ec8 2024-09-17T07:23:37.318 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi179.ecjhll smithi179 *:80 running (9m) 25s ago 9m 42.5M - 16.2.0 24ecd6d5f14c 253109ee0394 2024-09-17T07:23:37.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:37 smithi044 ceph-mon[58191]: osdmap e103: 8 total, 7 up, 8 in 2024-09-17T07:23:37.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:37 smithi044 ceph-mon[58191]: from='client.54706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:23:37.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:37 smithi044 ceph-mon[60356]: osdmap e103: 8 total, 7 up, 8 in 2024-09-17T07:23:37.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:37 smithi044 ceph-mon[60356]: from='client.54706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:23:37.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:37 smithi179 ceph-mon[50085]: osdmap e103: 8 total, 7 up, 8 in 2024-09-17T07:23:37.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:37 smithi179 ceph-mon[50085]: from='client.54706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:23:37.614 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:37 smithi179 podman[55728]: 2024-09-17 07:23:37.345465952 +0000 UTC m=+0.674231215 container remove b095455891799c184f3cdb9d8232caa8cd34d5cdf49919876dcd139847920165 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-17T07:23:37.614 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:37 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.5.service: Deactivated successfully. 2024-09-17T07:23:37.614 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:37 smithi179 systemd[1]: Stopped Ceph osd.5 for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:23:37.614 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:37 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.5.service: Consumed 1.127s CPU time. 2024-09-17T07:23:37.614 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:37 smithi179 systemd[1]: Starting Ceph osd.5 for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:23:37.719 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:23:37.719 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-09-17T07:23:37.719 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3 2024-09-17T07:23:37.719 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:23:37.719 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-09-17T07:23:37.719 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T07:23:37.719 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:23:37.719 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-09-17T07:23:37.719 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 2, 2024-09-17T07:23:37.720 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 5 2024-09-17T07:23:37.720 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:23:37.720 INFO:teuthology.orchestra.run.smithi044.stdout: "rgw": { 2024-09-17T07:23:37.720 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2024-09-17T07:23:37.720 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:23:37.720 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-09-17T07:23:37.720 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 6, 2024-09-17T07:23:37.720 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 10 2024-09-17T07:23:37.720 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-09-17T07:23:37.720 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:23:38.048 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:37 smithi179 podman[55830]: 2024-09-17 07:23:37.684564495 +0000 UTC m=+0.015738984 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:23:38.049 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:37 smithi179 podman[55830]: 2024-09-17 07:23:37.787925294 +0000 UTC m=+0.119099759 container create 25ecae30b056729b609e0de36c91973bd077a44865acb4ebf284508235c2b414 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5-activate, RELEASE=reef-19e2a2b, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T07:23:38.049 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:38 smithi179 podman[55830]: 2024-09-17 07:23:38.048892765 +0000 UTC m=+0.380067249 container init 25ecae30b056729b609e0de36c91973bd077a44865acb4ebf284508235c2b414 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5-activate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-17T07:23:38.059 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:23:38.059 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9", 2024-09-17T07:23:38.059 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-09-17T07:23:38.059 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T07:23:38.059 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [ 2024-09-17T07:23:38.059 INFO:teuthology.orchestra.run.smithi044.stdout: "mon", 2024-09-17T07:23:38.059 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr" 2024-09-17T07:23:38.059 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-09-17T07:23:38.059 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "10/23 daemons upgraded", 2024-09-17T07:23:38.059 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "Currently upgrading osd daemons", 2024-09-17T07:23:38.060 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-09-17T07:23:38.060 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:23:38.346 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:38 smithi179 podman[55830]: 2024-09-17 07:23:38.052238661 +0000 UTC m=+0.383413128 container start 25ecae30b056729b609e0de36c91973bd077a44865acb4ebf284508235c2b414 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5-activate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T07:23:38.347 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:38 smithi179 podman[55830]: 2024-09-17 07:23:38.140351678 +0000 UTC m=+0.471526157 container attach 25ecae30b056729b609e0de36c91973bd077a44865acb4ebf284508235c2b414 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5-activate, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T07:23:38.459 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_WARN 2 failed cephadm daemon(s); 1 osds down; Reduced data availability: 3 pgs inactive, 10 pgs peering 2024-09-17T07:23:38.459 INFO:teuthology.orchestra.run.smithi044.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T07:23:38.459 INFO:teuthology.orchestra.run.smithi044.stdout: daemon iscsi.foo.smithi044.elmtfk on smithi044 is in unknown state 2024-09-17T07:23:38.459 INFO:teuthology.orchestra.run.smithi044.stdout: daemon prometheus.a on smithi179 is in error state 2024-09-17T07:23:38.459 INFO:teuthology.orchestra.run.smithi044.stdout:[WRN] OSD_DOWN: 1 osds down 2024-09-17T07:23:38.459 INFO:teuthology.orchestra.run.smithi044.stdout: osd.5 (root=default,host=smithi179) is down 2024-09-17T07:23:38.460 INFO:teuthology.orchestra.run.smithi044.stdout:[WRN] PG_AVAILABILITY: Reduced data availability: 3 pgs inactive, 10 pgs peering 2024-09-17T07:23:38.460 INFO:teuthology.orchestra.run.smithi044.stdout: pg 2.3 is stuck peering for 9m, current state peering, last acting [2,7] 2024-09-17T07:23:38.460 INFO:teuthology.orchestra.run.smithi044.stdout: pg 2.12 is stuck peering for 9m, current state peering, last acting [3,7] 2024-09-17T07:23:38.460 INFO:teuthology.orchestra.run.smithi044.stdout: pg 2.16 is stuck peering for 65s, current state peering, last acting [6,2] 2024-09-17T07:23:38.460 INFO:teuthology.orchestra.run.smithi044.stdout: pg 3.5 is stuck peering for 9m, current state peering, last acting [3,2] 2024-09-17T07:23:38.460 INFO:teuthology.orchestra.run.smithi044.stdout: pg 3.c is stuck peering for 9m, current state peering, last acting [3,6] 2024-09-17T07:23:38.460 INFO:teuthology.orchestra.run.smithi044.stdout: pg 3.16 is stuck peering for 9m, current state peering, last acting [7,1] 2024-09-17T07:23:38.460 INFO:teuthology.orchestra.run.smithi044.stdout: pg 3.1d is stuck peering for 9m, current state peering, last acting [4,6] 2024-09-17T07:23:38.460 INFO:teuthology.orchestra.run.smithi044.stdout: pg 4.8 is stuck peering for 9m, current state peering, last acting [7,6] 2024-09-17T07:23:38.460 INFO:teuthology.orchestra.run.smithi044.stdout: pg 4.15 is stuck peering for 9m, current state peering, last acting [7,3] 2024-09-17T07:23:38.460 INFO:teuthology.orchestra.run.smithi044.stdout: pg 6.a is stuck peering for 2m, current state peering, last acting [6,0] 2024-09-17T07:23:38.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:38 smithi179 ceph-mon[50085]: from='client.54712 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:23:38.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:38 smithi179 ceph-mon[50085]: from='client.54718 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:23:38.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:38 smithi179 ceph-mon[50085]: pgmap v155: 161 pgs: 41 peering, 6 stale+active+clean, 114 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:23:38.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:38 smithi179 ceph-mon[50085]: Health check failed: Reduced data availability: 3 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2024-09-17T07:23:38.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:38 smithi179 ceph-mon[50085]: from='client.44838 172.21.15.44:0/739591320' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:23:38.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:38 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/4076726154' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:23:38.614 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:38 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5-activate[55845]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-17T07:23:38.615 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:38 smithi179 bash[55830]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-17T07:23:38.615 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:38 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5-activate[55845]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-17T07:23:38.615 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:38 smithi179 bash[55830]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-17T07:23:38.615 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:38 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5-activate[55845]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-17T07:23:38.615 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:38 smithi179 bash[55830]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-17T07:23:38.615 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:38 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5-activate[55845]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-17T07:23:38.615 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:38 smithi179 bash[55830]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-17T07:23:38.615 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:38 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5-activate[55845]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-09-17T07:23:38.615 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:38 smithi179 bash[55830]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-09-17T07:23:38.615 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:38 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5-activate[55845]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-17T07:23:38.615 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:38 smithi179 bash[55830]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-17T07:23:38.615 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:38 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5-activate[55845]: --> ceph-volume raw activate successful for osd ID: 5 2024-09-17T07:23:38.615 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:38 smithi179 bash[55830]: --> ceph-volume raw activate successful for osd ID: 5 2024-09-17T07:23:38.615 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:38 smithi179 podman[55830]: 2024-09-17 07:23:38.393963472 +0000 UTC m=+0.725137948 container died 25ecae30b056729b609e0de36c91973bd077a44865acb4ebf284508235c2b414 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5-activate, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, ceph=True, io.buildah.version=1.37.2) 2024-09-17T07:23:38.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:38 smithi044 ceph-mon[58191]: from='client.54712 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:23:38.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:38 smithi044 ceph-mon[58191]: from='client.54718 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:23:38.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:38 smithi044 ceph-mon[58191]: pgmap v155: 161 pgs: 41 peering, 6 stale+active+clean, 114 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:23:38.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:38 smithi044 ceph-mon[58191]: Health check failed: Reduced data availability: 3 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2024-09-17T07:23:38.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:38 smithi044 ceph-mon[58191]: from='client.44838 172.21.15.44:0/739591320' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:23:38.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:38 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/4076726154' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:23:38.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:38 smithi044 ceph-mon[60356]: from='client.54712 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:23:38.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:38 smithi044 ceph-mon[60356]: from='client.54718 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:23:38.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:38 smithi044 ceph-mon[60356]: pgmap v155: 161 pgs: 41 peering, 6 stale+active+clean, 114 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:23:38.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:38 smithi044 ceph-mon[60356]: Health check failed: Reduced data availability: 3 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2024-09-17T07:23:38.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:38 smithi044 ceph-mon[60356]: from='client.44838 172.21.15.44:0/739591320' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:23:38.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:38 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/4076726154' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:23:39.114 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:38 smithi179 podman[55963]: 2024-09-17 07:23:38.988055681 +0000 UTC m=+0.587200445 container remove 25ecae30b056729b609e0de36c91973bd077a44865acb4ebf284508235c2b414 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5-activate, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T07:23:39.444 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:39 smithi179 podman[56026]: 2024-09-17 07:23:39.095693237 +0000 UTC m=+0.015792520 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:23:39.444 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:39 smithi179 podman[56026]: 2024-09-17 07:23:39.197297529 +0000 UTC m=+0.117396828 container create 0258eb8ed0e71089ea84a1288783e16a1661df4b2d29a05838957a419f38dc34 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-17T07:23:39.444 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:39 smithi179 podman[56026]: 2024-09-17 07:23:39.44435396 +0000 UTC m=+0.364453225 container init 0258eb8ed0e71089ea84a1288783e16a1661df4b2d29a05838957a419f38dc34 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-17T07:23:39.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:39 smithi044 ceph-mon[58191]: from='client.44844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:23:39.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:39 smithi044 ceph-mon[60356]: from='client.44844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:23:39.863 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:39 smithi179 ceph-mon[50085]: from='client.44844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:23:39.864 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:39 smithi179 podman[56026]: 2024-09-17 07:23:39.447480343 +0000 UTC m=+0.367579600 container start 0258eb8ed0e71089ea84a1288783e16a1661df4b2d29a05838957a419f38dc34 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, ceph=True, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-17T07:23:39.864 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:39 smithi179 ceph-osd[56045]: -- 172.21.15.179:0/1362395386 <== mon.2 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5592fb5611e0 con 0x5592fc2e0000 2024-09-17T07:23:39.864 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:39 smithi179 bash[56026]: 0258eb8ed0e71089ea84a1288783e16a1661df4b2d29a05838957a419f38dc34 2024-09-17T07:23:39.864 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:39 smithi179 systemd[1]: Started Ceph osd.5 for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:23:40.298 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:39 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5[56041]: 2024-09-17T07:23:39.985+0000 7f5327ad4640 -1 Falling back to public interface 2024-09-17T07:23:40.569 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:40 smithi179 ceph-mon[50085]: pgmap v156: 161 pgs: 9 active+undersized, 41 peering, 1 stale+active+clean, 2 active+undersized+degraded, 108 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 14/999 objects degraded (1.401%) 2024-09-17T07:23:40.569 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:40 smithi179 ceph-mon[50085]: Health check failed: Degraded data redundancy: 14/999 objects degraded (1.401%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T07:23:40.569 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:40 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:40.569 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:40 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:40.569 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:40 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:23:40.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:40 smithi044 ceph-mon[58191]: pgmap v156: 161 pgs: 9 active+undersized, 41 peering, 1 stale+active+clean, 2 active+undersized+degraded, 108 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 14/999 objects degraded (1.401%) 2024-09-17T07:23:40.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:40 smithi044 ceph-mon[58191]: Health check failed: Degraded data redundancy: 14/999 objects degraded (1.401%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T07:23:40.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:40 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:40.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:40 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:40.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:40 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:23:40.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:40 smithi044 ceph-mon[60356]: pgmap v156: 161 pgs: 9 active+undersized, 41 peering, 1 stale+active+clean, 2 active+undersized+degraded, 108 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 14/999 objects degraded (1.401%) 2024-09-17T07:23:40.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:40 smithi044 ceph-mon[60356]: Health check failed: Degraded data redundancy: 14/999 objects degraded (1.401%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T07:23:40.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:40 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:40.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:40 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:40.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:40 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:23:41.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:41 smithi179 ceph-mon[50085]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 10 pgs peering) 2024-09-17T07:23:42.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:41 smithi044 ceph-mon[58191]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 10 pgs peering) 2024-09-17T07:23:42.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:41 smithi044 ceph-mon[60356]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 10 pgs peering) 2024-09-17T07:23:42.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:42 smithi179 ceph-mon[50085]: pgmap v157: 161 pgs: 37 active+undersized, 21 active+undersized+degraded, 103 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 118/999 objects degraded (11.812%) 2024-09-17T07:23:42.864 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5[56041]: 2024-09-17T07:23:42.706+0000 7f5327ad4640 -1 osd.5 0 read_superblock omap replica is missing. 2024-09-17T07:23:42.864 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5[56041]: 2024-09-17T07:23:42.730+0000 7f5327ad4640 -1 osd.5 101 log_to_monitors true 2024-09-17T07:23:43.020 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:42 smithi044 ceph-mon[58191]: pgmap v157: 161 pgs: 37 active+undersized, 21 active+undersized+degraded, 103 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 118/999 objects degraded (11.812%) 2024-09-17T07:23:43.020 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:42 smithi044 ceph-mon[60356]: pgmap v157: 161 pgs: 37 active+undersized, 21 active+undersized+degraded, 103 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 118/999 objects degraded (11.812%) 2024-09-17T07:23:44.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:43 smithi044 ceph-mon[58191]: from='osd.5 [v2:172.21.15.179:6808/2328615341,v1:172.21.15.179:6809/2328615341]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T07:23:44.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:43 smithi044 ceph-mon[58191]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T07:23:44.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:43 smithi044 ceph-mon[60356]: from='osd.5 [v2:172.21.15.179:6808/2328615341,v1:172.21.15.179:6809/2328615341]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T07:23:44.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:43 smithi044 ceph-mon[60356]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T07:23:44.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:43 smithi179 ceph-mon[50085]: from='osd.5 [v2:172.21.15.179:6808/2328615341,v1:172.21.15.179:6809/2328615341]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T07:23:44.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:43 smithi179 ceph-mon[50085]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T07:23:44.114 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:23:43 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5[56041]: 2024-09-17T07:23:43.659+0000 7f531f87e640 -1 osd.5 101 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T07:23:44.910 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:44 smithi044 ceph-mon[58191]: pgmap v158: 161 pgs: 37 active+undersized, 21 active+undersized+degraded, 103 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 118/999 objects degraded (11.812%) 2024-09-17T07:23:44.910 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:44 smithi044 ceph-mon[58191]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T07:23:44.910 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:44 smithi044 ceph-mon[58191]: osdmap e104: 8 total, 7 up, 8 in 2024-09-17T07:23:44.910 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:44 smithi044 ceph-mon[58191]: from='osd.5 [v2:172.21.15.179:6808/2328615341,v1:172.21.15.179:6809/2328615341]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:23:44.911 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:44 smithi044 ceph-mon[58191]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:23:44.911 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:44 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:44.911 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:44 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:44.911 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:44 smithi044 ceph-mon[58191]: from='client.54757 172.21.15.44:0/1689146740' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:23:44.911 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:44 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:44.911 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:44 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:44.911 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:44 smithi044 ceph-mon[60356]: pgmap v158: 161 pgs: 37 active+undersized, 21 active+undersized+degraded, 103 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 118/999 objects degraded (11.812%) 2024-09-17T07:23:44.911 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:44 smithi044 ceph-mon[60356]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T07:23:44.911 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:44 smithi044 ceph-mon[60356]: osdmap e104: 8 total, 7 up, 8 in 2024-09-17T07:23:44.911 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:44 smithi044 ceph-mon[60356]: from='osd.5 [v2:172.21.15.179:6808/2328615341,v1:172.21.15.179:6809/2328615341]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:23:44.911 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:44 smithi044 ceph-mon[60356]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:23:44.912 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:44 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:44.912 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:44 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:44.912 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:44 smithi044 ceph-mon[60356]: from='client.54757 172.21.15.44:0/1689146740' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:23:44.912 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:44 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:44.912 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:44 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:44.987 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:44 smithi179 ceph-mon[50085]: pgmap v158: 161 pgs: 37 active+undersized, 21 active+undersized+degraded, 103 active+clean; 604 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 118/999 objects degraded (11.812%) 2024-09-17T07:23:44.987 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:44 smithi179 ceph-mon[50085]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T07:23:44.987 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:44 smithi179 ceph-mon[50085]: osdmap e104: 8 total, 7 up, 8 in 2024-09-17T07:23:44.987 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:44 smithi179 ceph-mon[50085]: from='osd.5 [v2:172.21.15.179:6808/2328615341,v1:172.21.15.179:6809/2328615341]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:23:44.988 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:44 smithi179 ceph-mon[50085]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:23:44.988 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:44 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:44.988 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:44 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:44.988 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:44 smithi179 ceph-mon[50085]: from='client.54757 172.21.15.44:0/1689146740' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T07:23:44.988 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:44 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:44.988 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:44 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:46.011 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:45 smithi179 ceph-mon[50085]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T07:23:46.012 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:45 smithi179 ceph-mon[50085]: osd.5 [v2:172.21.15.179:6808/2328615341,v1:172.21.15.179:6809/2328615341] boot 2024-09-17T07:23:46.012 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:45 smithi179 ceph-mon[50085]: osdmap e105: 8 total, 8 up, 8 in 2024-09-17T07:23:46.012 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:45 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:23:46.012 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:45 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/2039173313' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:23:46.012 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:45 smithi179 ceph-mon[50085]: osdmap e106: 8 total, 8 up, 8 in 2024-09-17T07:23:46.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:45 smithi044 ceph-mon[60356]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T07:23:46.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:45 smithi044 ceph-mon[60356]: osd.5 [v2:172.21.15.179:6808/2328615341,v1:172.21.15.179:6809/2328615341] boot 2024-09-17T07:23:46.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:45 smithi044 ceph-mon[60356]: osdmap e105: 8 total, 8 up, 8 in 2024-09-17T07:23:46.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:45 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:23:46.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:45 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/2039173313' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:23:46.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:45 smithi044 ceph-mon[60356]: osdmap e106: 8 total, 8 up, 8 in 2024-09-17T07:23:46.077 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:45 smithi044 ceph-mon[58191]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T07:23:46.077 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:45 smithi044 ceph-mon[58191]: osd.5 [v2:172.21.15.179:6808/2328615341,v1:172.21.15.179:6809/2328615341] boot 2024-09-17T07:23:46.077 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:45 smithi044 ceph-mon[58191]: osdmap e105: 8 total, 8 up, 8 in 2024-09-17T07:23:46.077 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:45 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T07:23:46.077 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:45 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/2039173313' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: access denied 2024-09-17T07:23:46.077 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:45 smithi044 ceph-mon[58191]: osdmap e106: 8 total, 8 up, 8 in 2024-09-17T07:23:47.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:46 smithi044 ceph-mon[58191]: pgmap v161: 161 pgs: 5 peering, 35 active+undersized, 18 active+undersized+degraded, 103 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 102/999 objects degraded (10.210%) 2024-09-17T07:23:47.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:46 smithi044 ceph-mon[58191]: Health check update: Degraded data redundancy: 102/999 objects degraded (10.210%), 18 pgs degraded (PG_DEGRADED) 2024-09-17T07:23:47.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:46 smithi044 ceph-mon[60356]: pgmap v161: 161 pgs: 5 peering, 35 active+undersized, 18 active+undersized+degraded, 103 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 102/999 objects degraded (10.210%) 2024-09-17T07:23:47.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:46 smithi044 ceph-mon[60356]: Health check update: Degraded data redundancy: 102/999 objects degraded (10.210%), 18 pgs degraded (PG_DEGRADED) 2024-09-17T07:23:47.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:46 smithi179 ceph-mon[50085]: pgmap v161: 161 pgs: 5 peering, 35 active+undersized, 18 active+undersized+degraded, 103 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 102/999 objects degraded (10.210%) 2024-09-17T07:23:47.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:46 smithi179 ceph-mon[50085]: Health check update: Degraded data redundancy: 102/999 objects degraded (10.210%), 18 pgs degraded (PG_DEGRADED) 2024-09-17T07:23:48.574 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:23:48 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:23:48.205+0000 7f5a7ae47640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-09-17T07:23:49.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:48 smithi044 ceph-mon[60356]: pgmap v163: 161 pgs: 5 peering, 17 active+undersized, 10 active+undersized+degraded, 129 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 64/999 objects degraded (6.406%) 2024-09-17T07:23:49.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:48 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:49.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:48 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:49.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:48 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:23:49.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:48 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:23:49.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:48 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:49.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:48 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:23:49.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:48 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:23:49.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:48 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:23:49.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:48 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:23:49.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:48 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T07:23:49.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:48 smithi044 ceph-mon[58191]: pgmap v163: 161 pgs: 5 peering, 17 active+undersized, 10 active+undersized+degraded, 129 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 64/999 objects degraded (6.406%) 2024-09-17T07:23:49.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:48 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:49.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:48 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:49.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:48 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:23:49.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:48 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:23:49.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:48 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:49.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:48 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:23:49.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:48 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:23:49.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:48 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:23:49.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:48 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:23:49.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:48 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T07:23:49.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:48 smithi179 ceph-mon[50085]: pgmap v163: 161 pgs: 5 peering, 17 active+undersized, 10 active+undersized+degraded, 129 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 64/999 objects degraded (6.406%) 2024-09-17T07:23:49.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:48 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:49.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:48 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:49.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:48 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:23:49.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:48 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:23:49.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:48 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:49.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:48 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:23:49.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:48 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:23:49.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:48 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:23:49.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:48 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:23:49.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:48 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T07:23:50.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:49 smithi044 ceph-mon[58191]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:23:50.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:49 smithi044 ceph-mon[58191]: pgmap v164: 161 pgs: 5 peering, 17 active+undersized, 10 active+undersized+degraded, 129 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 222 B/s rd, 0 op/s; 64/999 objects degraded (6.406%) 2024-09-17T07:23:50.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:49 smithi044 ceph-mon[58191]: pgmap v165: 161 pgs: 5 peering, 17 active+undersized, 10 active+undersized+degraded, 129 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 230 B/s rd, 0 op/s; 64/999 objects degraded (6.406%) 2024-09-17T07:23:50.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:49 smithi044 ceph-mon[58191]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T07:23:50.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:49 smithi044 ceph-mon[58191]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-09-17T07:23:50.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:49 smithi044 ceph-mon[60356]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:23:50.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:49 smithi044 ceph-mon[60356]: pgmap v164: 161 pgs: 5 peering, 17 active+undersized, 10 active+undersized+degraded, 129 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 222 B/s rd, 0 op/s; 64/999 objects degraded (6.406%) 2024-09-17T07:23:50.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:49 smithi044 ceph-mon[60356]: pgmap v165: 161 pgs: 5 peering, 17 active+undersized, 10 active+undersized+degraded, 129 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 230 B/s rd, 0 op/s; 64/999 objects degraded (6.406%) 2024-09-17T07:23:50.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:49 smithi044 ceph-mon[60356]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T07:23:50.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:49 smithi044 ceph-mon[60356]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-09-17T07:23:50.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:49 smithi179 ceph-mon[50085]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:23:50.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:49 smithi179 ceph-mon[50085]: pgmap v164: 161 pgs: 5 peering, 17 active+undersized, 10 active+undersized+degraded, 129 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 222 B/s rd, 0 op/s; 64/999 objects degraded (6.406%) 2024-09-17T07:23:50.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:49 smithi179 ceph-mon[50085]: pgmap v165: 161 pgs: 5 peering, 17 active+undersized, 10 active+undersized+degraded, 129 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 230 B/s rd, 0 op/s; 64/999 objects degraded (6.406%) 2024-09-17T07:23:50.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:49 smithi179 ceph-mon[50085]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T07:23:50.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:49 smithi179 ceph-mon[50085]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-09-17T07:23:51.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:51 smithi044 ceph-mon[58191]: pgmap v166: 161 pgs: 161 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 188 B/s rd, 0 op/s 2024-09-17T07:23:51.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:51 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:51.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:51 smithi044 ceph-mon[58191]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 64/999 objects degraded (6.406%), 10 pgs degraded) 2024-09-17T07:23:51.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:51 smithi044 ceph-mon[60356]: pgmap v166: 161 pgs: 161 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 188 B/s rd, 0 op/s 2024-09-17T07:23:51.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:51 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:51.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:51 smithi044 ceph-mon[60356]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 64/999 objects degraded (6.406%), 10 pgs degraded) 2024-09-17T07:23:51.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:51 smithi179 ceph-mon[50085]: pgmap v166: 161 pgs: 161 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 188 B/s rd, 0 op/s 2024-09-17T07:23:51.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:51 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:23:51.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:51 smithi179 ceph-mon[50085]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 64/999 objects degraded (6.406%), 10 pgs degraded) 2024-09-17T07:23:53.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:53 smithi044 ceph-mon[58191]: pgmap v167: 161 pgs: 161 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 155 B/s rd, 0 op/s 2024-09-17T07:23:53.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:53 smithi044 ceph-mon[60356]: pgmap v167: 161 pgs: 161 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 155 B/s rd, 0 op/s 2024-09-17T07:23:53.613 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:53 smithi179 ceph-mon[50085]: pgmap v167: 161 pgs: 161 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 155 B/s rd, 0 op/s 2024-09-17T07:23:55.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:55 smithi044 ceph-mon[58191]: pgmap v168: 161 pgs: 161 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 121 B/s rd, 0 op/s 2024-09-17T07:23:55.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:55 smithi044 ceph-mon[60356]: pgmap v168: 161 pgs: 161 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 121 B/s rd, 0 op/s 2024-09-17T07:23:55.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:55 smithi179 ceph-mon[50085]: pgmap v168: 161 pgs: 161 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 121 B/s rd, 0 op/s 2024-09-17T07:23:57.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:57 smithi044 ceph-mon[60356]: pgmap v169: 161 pgs: 161 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:23:57.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:57 smithi044 ceph-mon[58191]: pgmap v169: 161 pgs: 161 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:23:57.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:57 smithi179 ceph-mon[50085]: pgmap v169: 161 pgs: 161 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:23:59.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:23:59 smithi044 ceph-mon[58191]: pgmap v170: 161 pgs: 161 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:23:59.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:23:59 smithi044 ceph-mon[60356]: pgmap v170: 161 pgs: 161 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:23:59.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:23:59 smithi179 ceph-mon[50085]: pgmap v170: 161 pgs: 161 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:24:01.574 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:01 smithi044 ceph-mon[60356]: pgmap v171: 161 pgs: 161 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:24:01.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:01 smithi044 ceph-mon[58191]: pgmap v171: 161 pgs: 161 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:24:01.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:01 smithi179 ceph-mon[50085]: pgmap v171: 161 pgs: 161 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:24:03.560 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:03 smithi179 ceph-mon[50085]: pgmap v172: 161 pgs: 161 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:24:03.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:03 smithi044 ceph-mon[58191]: pgmap v172: 161 pgs: 161 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:24:03.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:03 smithi044 ceph-mon[60356]: pgmap v172: 161 pgs: 161 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:24:04.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:04 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T07:24:04.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:04 smithi044 ceph-mon[58191]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T07:24:04.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:04 smithi044 ceph-mon[58191]: Upgrade: osd.6 is safe to restart 2024-09-17T07:24:04.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:04 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T07:24:04.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:04 smithi044 ceph-mon[60356]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T07:24:04.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:04 smithi044 ceph-mon[60356]: Upgrade: osd.6 is safe to restart 2024-09-17T07:24:04.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:04 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T07:24:04.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:04 smithi179 ceph-mon[50085]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T07:24:04.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:04 smithi179 ceph-mon[50085]: Upgrade: osd.6 is safe to restart 2024-09-17T07:24:05.959 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:05 smithi179 ceph-mon[50085]: pgmap v173: 161 pgs: 161 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:24:05.960 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:05 smithi179 ceph-mon[50085]: Upgrade: Updating osd.6 2024-09-17T07:24:05.960 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:05 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:05.960 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:05 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-17T07:24:05.960 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:05 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:24:05.960 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:05 smithi179 ceph-mon[50085]: Deploying daemon osd.6 on smithi179 2024-09-17T07:24:06.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:05 smithi044 ceph-mon[58191]: pgmap v173: 161 pgs: 161 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:24:06.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:05 smithi044 ceph-mon[58191]: Upgrade: Updating osd.6 2024-09-17T07:24:06.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:05 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:06.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:05 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-17T07:24:06.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:05 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:24:06.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:05 smithi044 ceph-mon[58191]: Deploying daemon osd.6 on smithi179 2024-09-17T07:24:06.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:05 smithi044 ceph-mon[60356]: pgmap v173: 161 pgs: 161 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:24:06.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:05 smithi044 ceph-mon[60356]: Upgrade: Updating osd.6 2024-09-17T07:24:06.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:05 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:06.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:05 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-17T07:24:06.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:05 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:24:06.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:05 smithi044 ceph-mon[60356]: Deploying daemon osd.6 on smithi179 2024-09-17T07:24:07.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:07 smithi179 ceph-mon[50085]: pgmap v174: 161 pgs: 161 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:24:07.864 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:07 smithi179 systemd[1]: Stopping Ceph osd.6 for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:24:07.865 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.6[31384]: debug 2024-09-17T07:24:07.656+0000 7f1e68dca700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-17T07:24:07.865 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.6[31384]: debug 2024-09-17T07:24:07.656+0000 7f1e68dca700 -1 osd.6 106 *** Got signal Terminated *** 2024-09-17T07:24:07.865 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.6[31384]: debug 2024-09-17T07:24:07.656+0000 7f1e68dca700 -1 osd.6 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T07:24:07.865 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:07 smithi179 podman[57917]: 2024-09-17 07:24:07.687805669 +0000 UTC m=+0.138899853 container died 7a5c30b68cbc492ed34239a8d6840bf45523d53644556b13e041bba63245378c (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.6, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, RELEASE=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Dimitri Savineau , CEPH_POINT_RELEASE=-16.2.0, org.label-schema.build-date=20201204) 2024-09-17T07:24:08.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:07 smithi044 ceph-mon[58191]: pgmap v174: 161 pgs: 161 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:24:08.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:07 smithi044 ceph-mon[60356]: pgmap v174: 161 pgs: 161 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:24:08.334 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:08 smithi179 podman[57917]: 2024-09-17 07:24:08.058040441 +0000 UTC m=+0.509134629 container cleanup 7a5c30b68cbc492ed34239a8d6840bf45523d53644556b13e041bba63245378c (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.6, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Dimitri Savineau , org.label-schema.schema-version=1.0, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, RELEASE=HEAD, org.label-schema.build-date=20201204, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-16.2.0) 2024-09-17T07:24:08.335 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:08 smithi179 bash[57917]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.6 2024-09-17T07:24:08.614 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:08 smithi179 podman[57930]: 2024-09-17 07:24:08.334493597 +0000 UTC m=+0.646724197 container remove 7a5c30b68cbc492ed34239a8d6840bf45523d53644556b13e041bba63245378c (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.6, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, RELEASE=HEAD, org.label-schema.build-date=20201204, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-16.2.0, maintainer=Dimitri Savineau , org.label-schema.name=CentOS Base Image, org.label-schema.schema-version=1.0) 2024-09-17T07:24:08.838 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-09-17T07:24:09.008 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:08 smithi179 ceph-mon[50085]: osd.6 reported immediately failed by osd.1 2024-09-17T07:24:09.008 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:08 smithi179 ceph-mon[50085]: osd.6 failed (root=default,host=smithi179) (connection refused reported by osd.1) 2024-09-17T07:24:09.008 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:08 smithi179 ceph-mon[50085]: osd.6 reported immediately failed by osd.1 2024-09-17T07:24:09.008 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:08 smithi179 ceph-mon[50085]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T07:24:09.008 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:08 smithi179 ceph-mon[50085]: osdmap e107: 8 total, 7 up, 8 in 2024-09-17T07:24:09.009 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:08 smithi179 podman[58044]: 2024-09-17 07:24:08.691739877 +0000 UTC m=+0.016874581 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:24:09.009 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:08 smithi179 podman[58044]: 2024-09-17 07:24:08.902979141 +0000 UTC m=+0.228113826 container create 028795ee7cef65d65ec905ccd9b343545feaa72a767be97509d1020c6ce966d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T07:24:09.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:08 smithi044 ceph-mon[58191]: osd.6 reported immediately failed by osd.1 2024-09-17T07:24:09.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:08 smithi044 ceph-mon[58191]: osd.6 failed (root=default,host=smithi179) (connection refused reported by osd.1) 2024-09-17T07:24:09.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:08 smithi044 ceph-mon[58191]: osd.6 reported immediately failed by osd.1 2024-09-17T07:24:09.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:08 smithi044 ceph-mon[58191]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T07:24:09.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:08 smithi044 ceph-mon[58191]: osdmap e107: 8 total, 7 up, 8 in 2024-09-17T07:24:09.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:08 smithi044 ceph-mon[60356]: osd.6 reported immediately failed by osd.1 2024-09-17T07:24:09.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:08 smithi044 ceph-mon[60356]: osd.6 failed (root=default,host=smithi179) (connection refused reported by osd.1) 2024-09-17T07:24:09.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:08 smithi044 ceph-mon[60356]: osd.6 reported immediately failed by osd.1 2024-09-17T07:24:09.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:08 smithi044 ceph-mon[60356]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T07:24:09.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:08 smithi044 ceph-mon[60356]: osdmap e107: 8 total, 7 up, 8 in 2024-09-17T07:24:09.310 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:09 smithi179 podman[58044]: 2024-09-17 07:24:09.122389364 +0000 UTC m=+0.447524056 container init 028795ee7cef65d65ec905ccd9b343545feaa72a767be97509d1020c6ce966d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True) 2024-09-17T07:24:09.310 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:09 smithi179 podman[58044]: 2024-09-17 07:24:09.125696351 +0000 UTC m=+0.450831040 container start 028795ee7cef65d65ec905ccd9b343545feaa72a767be97509d1020c6ce966d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6-deactivate, ceph=True, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2) 2024-09-17T07:24:09.310 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:09 smithi179 podman[58044]: 2024-09-17 07:24:09.222986677 +0000 UTC m=+0.548121400 container attach 028795ee7cef65d65ec905ccd9b343545feaa72a767be97509d1020c6ce966d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-17T07:24:09.499 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T07:24:09.499 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.a smithi044 *:9093,9094 running (5m) 75s ago 10m 18.9M - 0.20.0 0881eb8f169f 0473d4b023ad 2024-09-17T07:24:09.499 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.a smithi179 *:3000 running (5m) 25s ago 10m 33.2M - 6.7.4 557c83e11646 43ab0e03dcac 2024-09-17T07:24:09.499 INFO:teuthology.orchestra.run.smithi044.stdout:iscsi.foo.smithi044.elmtfk smithi044 unknown 75s ago 9m - - 2024-09-17T07:24:09.499 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.x smithi179 *:8443,8765 running (4m) 25s ago 14m 438M - 18.2.4-855-g19e2a2be 980a0bfcd684 66b52dc26022 2024-09-17T07:24:09.499 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.y smithi044 *:8443,8765 running (5m) 75s ago 15m 482M - 18.2.4-855-g19e2a2be 980a0bfcd684 8080dd318a1c 2024-09-17T07:24:09.499 INFO:teuthology.orchestra.run.smithi044.stdout:mon.a smithi044 running (4m) 75s ago 16m 48.7M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 c3293e8cdbba 2024-09-17T07:24:09.499 INFO:teuthology.orchestra.run.smithi044.stdout:mon.b smithi179 running (3m) 25s ago 14m 44.6M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 ade749759184 2024-09-17T07:24:09.499 INFO:teuthology.orchestra.run.smithi044.stdout:mon.c smithi044 running (4m) 75s ago 15m 42.0M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 02f85d31e0bd 2024-09-17T07:24:09.500 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.a smithi044 *:9100 running (6m) 75s ago 10m 17.1M - 0.18.1 e5a616e4b9cf 8a2cde5bd9ed 2024-09-17T07:24:09.500 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.b smithi179 *:9100 running (5m) 25s ago 11m 8929k - 0.18.1 e5a616e4b9cf 04c63b5cb524 2024-09-17T07:24:09.500 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (2m) 75s ago 14m 52.4M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 3ba47a4e2e95 2024-09-17T07:24:09.500 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (2m) 75s ago 13m 50.5M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 7655e43d9944 2024-09-17T07:24:09.500 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (102s) 75s ago 13m 49.0M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 43a48289eada 2024-09-17T07:24:09.500 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi044 running (83s) 75s ago 13m 11.5M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 b4323c928f39 2024-09-17T07:24:09.500 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi179 running (62s) 25s ago 13m 46.8M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 917adc8feb0b 2024-09-17T07:24:09.500 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi179 running (30s) 25s ago 12m 11.0M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 0258eb8ed0e7 2024-09-17T07:24:09.500 INFO:teuthology.orchestra.run.smithi044.stdout:osd.6 smithi179 running (12m) 25s ago 12m 53.0M 4096M 16.2.0 24ecd6d5f14c 7a5c30b68cbc 2024-09-17T07:24:09.500 INFO:teuthology.orchestra.run.smithi044.stdout:osd.7 smithi179 running (12m) 25s ago 12m 57.8M 4096M 16.2.0 24ecd6d5f14c b0d186fbf8a6 2024-09-17T07:24:09.500 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.a smithi179 *:9095 error 25s ago 11m - - 2024-09-17T07:24:09.501 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi044.nhyhnh smithi044 *:8000 running (9m) 75s ago 9m 39.1M - 16.2.0 24ecd6d5f14c 9b455479614f 2024-09-17T07:24:09.501 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi179.tqlzyy smithi179 *:8000 running (9m) 25s ago 9m 42.0M - 16.2.0 24ecd6d5f14c c8d1693241ce 2024-09-17T07:24:09.501 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi044.vnufiz smithi044 *:80 running (9m) 75s ago 9m 39.2M - 16.2.0 24ecd6d5f14c 6eae2bff5ec8 2024-09-17T07:24:09.501 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi179.ecjhll smithi179 *:80 running (9m) 25s ago 9m 42.7M - 16.2.0 24ecd6d5f14c 253109ee0394 2024-09-17T07:24:09.613 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:09 smithi179 podman[58044]: 2024-09-17 07:24:09.310399774 +0000 UTC m=+0.635534472 container died 028795ee7cef65d65ec905ccd9b343545feaa72a767be97509d1020c6ce966d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6-deactivate, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-17T07:24:09.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:09 smithi044 ceph-mon[58191]: pgmap v176: 161 pgs: 13 stale+active+clean, 148 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:24:09.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:09 smithi044 ceph-mon[58191]: osdmap e108: 8 total, 7 up, 8 in 2024-09-17T07:24:09.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:09 smithi044 ceph-mon[58191]: from='client.44883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:24:09.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:09 smithi044 ceph-mon[60356]: pgmap v176: 161 pgs: 13 stale+active+clean, 148 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:24:09.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:09 smithi044 ceph-mon[60356]: osdmap e108: 8 total, 7 up, 8 in 2024-09-17T07:24:09.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:09 smithi044 ceph-mon[60356]: from='client.44883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:24:09.903 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:24:09.903 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-09-17T07:24:09.903 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3 2024-09-17T07:24:09.903 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:24:09.904 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-09-17T07:24:09.904 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T07:24:09.904 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:24:09.904 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-09-17T07:24:09.904 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 1, 2024-09-17T07:24:09.904 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 6 2024-09-17T07:24:09.904 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:24:09.904 INFO:teuthology.orchestra.run.smithi044.stdout: "rgw": { 2024-09-17T07:24:09.904 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2024-09-17T07:24:09.904 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:24:09.904 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-09-17T07:24:09.904 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 5, 2024-09-17T07:24:09.904 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 11 2024-09-17T07:24:09.905 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-09-17T07:24:09.905 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:24:10.045 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:09 smithi179 ceph-mon[50085]: pgmap v176: 161 pgs: 13 stale+active+clean, 148 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:24:10.045 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:09 smithi179 ceph-mon[50085]: osdmap e108: 8 total, 7 up, 8 in 2024-09-17T07:24:10.045 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:09 smithi179 ceph-mon[50085]: from='client.44883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:24:10.242 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:24:10.242 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9", 2024-09-17T07:24:10.242 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-09-17T07:24:10.242 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T07:24:10.242 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [ 2024-09-17T07:24:10.242 INFO:teuthology.orchestra.run.smithi044.stdout: "mon", 2024-09-17T07:24:10.242 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr" 2024-09-17T07:24:10.242 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-09-17T07:24:10.242 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "11/23 daemons upgraded", 2024-09-17T07:24:10.242 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "Currently upgrading osd daemons", 2024-09-17T07:24:10.243 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-09-17T07:24:10.243 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:24:10.364 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:10 smithi179 podman[58068]: 2024-09-17 07:24:10.045218859 +0000 UTC m=+0.728699090 container remove 028795ee7cef65d65ec905ccd9b343545feaa72a767be97509d1020c6ce966d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, GIT_CLEAN=True) 2024-09-17T07:24:10.364 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:10 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.6.service: Deactivated successfully. 2024-09-17T07:24:10.364 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:10 smithi179 systemd[1]: Stopped Ceph osd.6 for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:24:10.364 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:10 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.6.service: Consumed 1.136s CPU time. 2024-09-17T07:24:10.364 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:10 smithi179 systemd[1]: Starting Ceph osd.6 for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:24:10.640 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_WARN 2 failed cephadm daemon(s); 1 osds down 2024-09-17T07:24:10.640 INFO:teuthology.orchestra.run.smithi044.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T07:24:10.640 INFO:teuthology.orchestra.run.smithi044.stdout: daemon iscsi.foo.smithi044.elmtfk on smithi044 is in unknown state 2024-09-17T07:24:10.641 INFO:teuthology.orchestra.run.smithi044.stdout: daemon prometheus.a on smithi179 is in error state 2024-09-17T07:24:10.641 INFO:teuthology.orchestra.run.smithi044.stdout:[WRN] OSD_DOWN: 1 osds down 2024-09-17T07:24:10.641 INFO:teuthology.orchestra.run.smithi044.stdout: osd.6 (root=default,host=smithi179) is down 2024-09-17T07:24:10.748 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:10 smithi179 podman[58165]: 2024-09-17 07:24:10.391115911 +0000 UTC m=+0.016288752 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:24:10.748 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:10 smithi179 podman[58165]: 2024-09-17 07:24:10.513250193 +0000 UTC m=+0.138423060 container create 5d4bbe398b2072387322a7132feb3c3f6fd94db8111c83e6159a8fe03db201b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6-activate, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-17T07:24:11.061 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:10 smithi179 ceph-mon[50085]: from='client.54775 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:24:11.061 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:10 smithi179 ceph-mon[50085]: from='client.54781 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:24:11.061 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:10 smithi179 ceph-mon[50085]: from='client.44901 172.21.15.44:0/3376242892' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:24:11.061 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:10 smithi179 ceph-mon[50085]: from='client.54799 172.21.15.44:0/3976171201' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:24:11.062 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:10 smithi179 podman[58165]: 2024-09-17 07:24:10.7654999 +0000 UTC m=+0.390672738 container init 5d4bbe398b2072387322a7132feb3c3f6fd94db8111c83e6159a8fe03db201b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6-activate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-17T07:24:11.062 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:10 smithi179 podman[58165]: 2024-09-17 07:24:10.769269444 +0000 UTC m=+0.394442272 container start 5d4bbe398b2072387322a7132feb3c3f6fd94db8111c83e6159a8fe03db201b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6-activate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, ceph=True) 2024-09-17T07:24:11.062 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:10 smithi179 podman[58165]: 2024-09-17 07:24:10.849279508 +0000 UTC m=+0.474452334 container attach 5d4bbe398b2072387322a7132feb3c3f6fd94db8111c83e6159a8fe03db201b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6-activate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-17T07:24:11.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:10 smithi044 ceph-mon[58191]: from='client.54775 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:24:11.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:10 smithi044 ceph-mon[58191]: from='client.54781 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:24:11.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:10 smithi044 ceph-mon[58191]: from='client.44901 172.21.15.44:0/3376242892' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:24:11.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:10 smithi044 ceph-mon[58191]: from='client.54799 172.21.15.44:0/3976171201' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:24:11.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:10 smithi044 ceph-mon[60356]: from='client.54775 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:24:11.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:10 smithi044 ceph-mon[60356]: from='client.54781 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:24:11.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:10 smithi044 ceph-mon[60356]: from='client.44901 172.21.15.44:0/3376242892' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:24:11.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:10 smithi044 ceph-mon[60356]: from='client.54799 172.21.15.44:0/3976171201' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:24:11.364 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:11 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6-activate[58180]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-17T07:24:11.364 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:11 smithi179 bash[58165]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-17T07:24:11.364 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:11 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6-activate[58180]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-17T07:24:11.365 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:11 smithi179 bash[58165]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-17T07:24:11.365 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:11 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6-activate[58180]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-17T07:24:11.365 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:11 smithi179 bash[58165]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-17T07:24:11.365 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:11 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6-activate[58180]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-17T07:24:11.365 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:11 smithi179 bash[58165]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-17T07:24:11.365 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:11 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6-activate[58180]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-09-17T07:24:11.365 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:11 smithi179 bash[58165]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-09-17T07:24:11.365 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:11 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6-activate[58180]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-17T07:24:11.365 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:11 smithi179 bash[58165]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-17T07:24:11.365 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:11 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6-activate[58180]: --> ceph-volume raw activate successful for osd ID: 6 2024-09-17T07:24:11.365 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:11 smithi179 bash[58165]: --> ceph-volume raw activate successful for osd ID: 6 2024-09-17T07:24:11.365 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:11 smithi179 podman[58165]: 2024-09-17 07:24:11.10943638 +0000 UTC m=+0.734609217 container died 5d4bbe398b2072387322a7132feb3c3f6fd94db8111c83e6159a8fe03db201b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6-activate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T07:24:11.749 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:11 smithi179 podman[58298]: 2024-09-17 07:24:11.678897531 +0000 UTC m=+0.563314910 container remove 5d4bbe398b2072387322a7132feb3c3f6fd94db8111c83e6159a8fe03db201b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6-activate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b) 2024-09-17T07:24:12.017 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:11 smithi179 ceph-mon[50085]: pgmap v178: 161 pgs: 2 active+undersized, 14 peering, 8 stale+active+clean, 1 active+undersized+degraded, 136 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 10/999 objects degraded (1.001%) 2024-09-17T07:24:12.018 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:11 smithi179 ceph-mon[50085]: from='client.44907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:24:12.018 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:11 smithi179 ceph-mon[50085]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-17T07:24:12.018 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:11 smithi179 ceph-mon[50085]: Health check failed: Degraded data redundancy: 10/999 objects degraded (1.001%), 1 pg degraded (PG_DEGRADED) 2024-09-17T07:24:12.018 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:11 smithi179 podman[58359]: 2024-09-17 07:24:11.788434344 +0000 UTC m=+0.015782519 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:24:12.018 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:11 smithi179 podman[58359]: 2024-09-17 07:24:11.913415677 +0000 UTC m=+0.140763854 container create 22baeb2f62cbca44f853c7ce856278e74fd3146bbd2bc472b1a1fe6740989d13 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T07:24:12.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:11 smithi044 ceph-mon[58191]: pgmap v178: 161 pgs: 2 active+undersized, 14 peering, 8 stale+active+clean, 1 active+undersized+degraded, 136 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 10/999 objects degraded (1.001%) 2024-09-17T07:24:12.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:11 smithi044 ceph-mon[58191]: from='client.44907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:24:12.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:11 smithi044 ceph-mon[58191]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-17T07:24:12.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:11 smithi044 ceph-mon[58191]: Health check failed: Degraded data redundancy: 10/999 objects degraded (1.001%), 1 pg degraded (PG_DEGRADED) 2024-09-17T07:24:12.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:11 smithi044 ceph-mon[60356]: pgmap v178: 161 pgs: 2 active+undersized, 14 peering, 8 stale+active+clean, 1 active+undersized+degraded, 136 active+clean; 604 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 10/999 objects degraded (1.001%) 2024-09-17T07:24:12.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:11 smithi044 ceph-mon[60356]: from='client.44907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:24:12.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:11 smithi044 ceph-mon[60356]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-17T07:24:12.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:11 smithi044 ceph-mon[60356]: Health check failed: Degraded data redundancy: 10/999 objects degraded (1.001%), 1 pg degraded (PG_DEGRADED) 2024-09-17T07:24:12.364 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:12 smithi179 podman[58359]: 2024-09-17 07:24:12.13290455 +0000 UTC m=+0.360252726 container init 22baeb2f62cbca44f853c7ce856278e74fd3146bbd2bc472b1a1fe6740989d13 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD) 2024-09-17T07:24:12.364 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:12 smithi179 podman[58359]: 2024-09-17 07:24:12.136176217 +0000 UTC m=+0.363524395 container start 22baeb2f62cbca44f853c7ce856278e74fd3146bbd2bc472b1a1fe6740989d13 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-17T07:24:12.364 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:12 smithi179 ceph-osd[58378]: -- 172.21.15.179:0/3846383828 <== mon.2 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55def3bc31e0 con 0x55def4942400 2024-09-17T07:24:12.364 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:12 smithi179 bash[58359]: 22baeb2f62cbca44f853c7ce856278e74fd3146bbd2bc472b1a1fe6740989d13 2024-09-17T07:24:12.364 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:12 smithi179 systemd[1]: Started Ceph osd.6 for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:24:12.979 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:12 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:24:12.673+0000 7f2ffcb1f640 -1 Falling back to public interface 2024-09-17T07:24:13.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:13 smithi044 ceph-mon[58191]: pgmap v179: 161 pgs: 28 active+undersized, 14 peering, 11 active+undersized+degraded, 108 active+clean; 604 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 88/999 objects degraded (8.809%) 2024-09-17T07:24:13.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:13 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:13.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:13 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:13.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:13 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:24:13.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:13 smithi044 ceph-mon[60356]: pgmap v179: 161 pgs: 28 active+undersized, 14 peering, 11 active+undersized+degraded, 108 active+clean; 604 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 88/999 objects degraded (8.809%) 2024-09-17T07:24:13.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:13 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:13.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:13 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:13.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:13 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:24:13.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:13 smithi179 ceph-mon[50085]: pgmap v179: 161 pgs: 28 active+undersized, 14 peering, 11 active+undersized+degraded, 108 active+clean; 604 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 88/999 objects degraded (8.809%) 2024-09-17T07:24:13.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:13 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:13.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:13 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:13.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:13 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:24:14.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:14 smithi179 ceph-mon[50085]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-17T07:24:14.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:14 smithi044 ceph-mon[58191]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-17T07:24:14.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:14 smithi044 ceph-mon[60356]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-17T07:24:15.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:15 smithi179 ceph-mon[50085]: pgmap v180: 161 pgs: 35 active+undersized, 18 active+undersized+degraded, 108 active+clean; 604 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 122/999 objects degraded (12.212%) 2024-09-17T07:24:15.614 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:24:15.392+0000 7f2ffcb1f640 -1 osd.6 0 read_superblock omap replica is missing. 2024-09-17T07:24:15.614 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:24:15.430+0000 7f2ffcb1f640 -1 osd.6 106 log_to_monitors true 2024-09-17T07:24:15.726 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:15 smithi044 ceph-mon[58191]: pgmap v180: 161 pgs: 35 active+undersized, 18 active+undersized+degraded, 108 active+clean; 604 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 122/999 objects degraded (12.212%) 2024-09-17T07:24:15.726 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:15 smithi044 ceph-mon[60356]: pgmap v180: 161 pgs: 35 active+undersized, 18 active+undersized+degraded, 108 active+clean; 604 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 122/999 objects degraded (12.212%) 2024-09-17T07:24:16.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:16 smithi044 ceph-mon[58191]: from='osd.6 [v2:172.21.15.179:6816/1281752871,v1:172.21.15.179:6817/1281752871]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T07:24:16.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:16 smithi044 ceph-mon[58191]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T07:24:16.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:16 smithi044 ceph-mon[60356]: from='osd.6 [v2:172.21.15.179:6816/1281752871,v1:172.21.15.179:6817/1281752871]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T07:24:16.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:16 smithi044 ceph-mon[60356]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T07:24:16.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:16 smithi179 ceph-mon[50085]: from='osd.6 [v2:172.21.15.179:6816/1281752871,v1:172.21.15.179:6817/1281752871]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T07:24:16.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:16 smithi179 ceph-mon[50085]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T07:24:16.614 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:24:16 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:24:16.374+0000 7f2ff48c9640 -1 osd.6 106 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T07:24:17.582 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:17 smithi179 ceph-mon[50085]: pgmap v181: 161 pgs: 35 active+undersized, 18 active+undersized+degraded, 108 active+clean; 604 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 4 op/s; 122/999 objects degraded (12.212%) 2024-09-17T07:24:17.583 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:17 smithi179 ceph-mon[50085]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-17T07:24:17.583 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:17 smithi179 ceph-mon[50085]: osdmap e109: 8 total, 7 up, 8 in 2024-09-17T07:24:17.583 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:17 smithi179 ceph-mon[50085]: from='osd.6 [v2:172.21.15.179:6816/1281752871,v1:172.21.15.179:6817/1281752871]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:24:17.583 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:17 smithi179 ceph-mon[50085]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:24:17.583 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:17 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:17.583 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:17 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:17.583 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:17 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:17.583 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:17 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:17.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:17 smithi044 ceph-mon[58191]: pgmap v181: 161 pgs: 35 active+undersized, 18 active+undersized+degraded, 108 active+clean; 604 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 4 op/s; 122/999 objects degraded (12.212%) 2024-09-17T07:24:17.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:17 smithi044 ceph-mon[58191]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-17T07:24:17.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:17 smithi044 ceph-mon[58191]: osdmap e109: 8 total, 7 up, 8 in 2024-09-17T07:24:17.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:17 smithi044 ceph-mon[58191]: from='osd.6 [v2:172.21.15.179:6816/1281752871,v1:172.21.15.179:6817/1281752871]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:24:17.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:17 smithi044 ceph-mon[58191]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:24:17.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:17 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:17.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:17 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:17.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:17 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:17.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:17 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:17.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:17 smithi044 ceph-mon[60356]: pgmap v181: 161 pgs: 35 active+undersized, 18 active+undersized+degraded, 108 active+clean; 604 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 4 op/s; 122/999 objects degraded (12.212%) 2024-09-17T07:24:17.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:17 smithi044 ceph-mon[60356]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-17T07:24:17.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:17 smithi044 ceph-mon[60356]: osdmap e109: 8 total, 7 up, 8 in 2024-09-17T07:24:17.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:17 smithi044 ceph-mon[60356]: from='osd.6 [v2:172.21.15.179:6816/1281752871,v1:172.21.15.179:6817/1281752871]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:24:17.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:17 smithi044 ceph-mon[60356]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:24:17.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:17 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:17.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:17 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:17.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:17 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:17.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:17 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:18.484 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:18 smithi179 ceph-mon[50085]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T07:24:18.484 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:18 smithi179 ceph-mon[50085]: osd.6 [v2:172.21.15.179:6816/1281752871,v1:172.21.15.179:6817/1281752871] boot 2024-09-17T07:24:18.484 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:18 smithi179 ceph-mon[50085]: osdmap e110: 8 total, 8 up, 8 in 2024-09-17T07:24:18.484 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:18 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:24:18.485 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:18 smithi179 ceph-mon[50085]: osdmap e111: 8 total, 8 up, 8 in 2024-09-17T07:24:18.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:18 smithi044 ceph-mon[60356]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T07:24:18.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:18 smithi044 ceph-mon[60356]: osd.6 [v2:172.21.15.179:6816/1281752871,v1:172.21.15.179:6817/1281752871] boot 2024-09-17T07:24:18.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:18 smithi044 ceph-mon[60356]: osdmap e110: 8 total, 8 up, 8 in 2024-09-17T07:24:18.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:18 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:24:18.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:18 smithi044 ceph-mon[60356]: osdmap e111: 8 total, 8 up, 8 in 2024-09-17T07:24:18.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:18 smithi044 ceph-mon[58191]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T07:24:18.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:18 smithi044 ceph-mon[58191]: osd.6 [v2:172.21.15.179:6816/1281752871,v1:172.21.15.179:6817/1281752871] boot 2024-09-17T07:24:18.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:18 smithi044 ceph-mon[58191]: osdmap e110: 8 total, 8 up, 8 in 2024-09-17T07:24:18.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:18 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T07:24:18.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:18 smithi044 ceph-mon[58191]: osdmap e111: 8 total, 8 up, 8 in 2024-09-17T07:24:19.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:19 smithi044 ceph-mon[58191]: pgmap v184: 161 pgs: 35 active+undersized, 18 active+undersized+degraded, 108 active+clean; 604 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 6 op/s; 122/999 objects degraded (12.212%) 2024-09-17T07:24:19.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:19 smithi044 ceph-mon[58191]: Health check update: Degraded data redundancy: 122/999 objects degraded (12.212%), 18 pgs degraded (PG_DEGRADED) 2024-09-17T07:24:19.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:19 smithi044 ceph-mon[60356]: pgmap v184: 161 pgs: 35 active+undersized, 18 active+undersized+degraded, 108 active+clean; 604 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 6 op/s; 122/999 objects degraded (12.212%) 2024-09-17T07:24:19.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:19 smithi044 ceph-mon[60356]: Health check update: Degraded data redundancy: 122/999 objects degraded (12.212%), 18 pgs degraded (PG_DEGRADED) 2024-09-17T07:24:19.856 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:19 smithi179 ceph-mon[50085]: pgmap v184: 161 pgs: 35 active+undersized, 18 active+undersized+degraded, 108 active+clean; 604 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 6 op/s; 122/999 objects degraded (12.212%) 2024-09-17T07:24:19.856 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:19 smithi179 ceph-mon[50085]: Health check update: Degraded data redundancy: 122/999 objects degraded (12.212%), 18 pgs degraded (PG_DEGRADED) 2024-09-17T07:24:20.824 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:24:20 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y[53332]: 2024-09-17T07:24:20.511+0000 7f5a7ae47640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-09-17T07:24:21.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:21 smithi044 ceph-mon[58191]: pgmap v186: 161 pgs: 28 active+undersized, 13 active+undersized+degraded, 120 active+clean; 604 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 105 op/s; 90/999 objects degraded (9.009%); 0 B/s, 2 objects/s recovering 2024-09-17T07:24:21.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:21 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:21.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:21 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:21.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:21 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:21.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:21 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:24:21.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:21 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:24:21.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:21 smithi044 ceph-mon[58191]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:24:21.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:21 smithi044 ceph-mon[58191]: pgmap v187: 161 pgs: 28 active+undersized, 13 active+undersized+degraded, 120 active+clean; 604 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 136 op/s; 90/999 objects degraded (9.009%); 0 B/s, 3 objects/s recovering 2024-09-17T07:24:21.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:21 smithi044 ceph-mon[58191]: pgmap v188: 161 pgs: 28 active+undersized, 13 active+undersized+degraded, 120 active+clean; 604 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 145 op/s; 90/999 objects degraded (9.009%); 0 B/s, 3 objects/s recovering 2024-09-17T07:24:21.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:21 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:21.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:21 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:24:21.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:21 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:24:21.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:21 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:24:21.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:21 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:24:21.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:21 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T07:24:21.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:21 smithi044 ceph-mon[58191]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T07:24:21.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:21 smithi044 ceph-mon[58191]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-09-17T07:24:21.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:21 smithi044 ceph-mon[60356]: pgmap v186: 161 pgs: 28 active+undersized, 13 active+undersized+degraded, 120 active+clean; 604 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 105 op/s; 90/999 objects degraded (9.009%); 0 B/s, 2 objects/s recovering 2024-09-17T07:24:21.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:21 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:21.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:21 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:21.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:21 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:21.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:21 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:24:21.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:21 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:24:21.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:21 smithi044 ceph-mon[60356]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:24:21.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:21 smithi044 ceph-mon[60356]: pgmap v187: 161 pgs: 28 active+undersized, 13 active+undersized+degraded, 120 active+clean; 604 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 136 op/s; 90/999 objects degraded (9.009%); 0 B/s, 3 objects/s recovering 2024-09-17T07:24:21.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:21 smithi044 ceph-mon[60356]: pgmap v188: 161 pgs: 28 active+undersized, 13 active+undersized+degraded, 120 active+clean; 604 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 145 op/s; 90/999 objects degraded (9.009%); 0 B/s, 3 objects/s recovering 2024-09-17T07:24:21.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:21 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:21.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:21 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:24:21.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:21 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:24:21.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:21 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:24:21.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:21 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:24:21.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:21 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T07:24:21.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:21 smithi044 ceph-mon[60356]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T07:24:21.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:21 smithi044 ceph-mon[60356]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-09-17T07:24:21.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:21 smithi179 ceph-mon[50085]: pgmap v186: 161 pgs: 28 active+undersized, 13 active+undersized+degraded, 120 active+clean; 604 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 105 op/s; 90/999 objects degraded (9.009%); 0 B/s, 2 objects/s recovering 2024-09-17T07:24:21.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:21 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:21.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:21 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:21.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:21 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:21.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:21 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:24:21.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:21 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:24:21.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:21 smithi179 ceph-mon[50085]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:24:21.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:21 smithi179 ceph-mon[50085]: pgmap v187: 161 pgs: 28 active+undersized, 13 active+undersized+degraded, 120 active+clean; 604 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 136 op/s; 90/999 objects degraded (9.009%); 0 B/s, 3 objects/s recovering 2024-09-17T07:24:21.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:21 smithi179 ceph-mon[50085]: pgmap v188: 161 pgs: 28 active+undersized, 13 active+undersized+degraded, 120 active+clean; 604 KiB data, 267 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 145 op/s; 90/999 objects degraded (9.009%); 0 B/s, 3 objects/s recovering 2024-09-17T07:24:21.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:21 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:21.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:21 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:24:21.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:21 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:24:21.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:21 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:24:21.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:21 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:24:21.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:21 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T07:24:21.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:21 smithi179 ceph-mon[50085]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T07:24:21.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:21 smithi179 ceph-mon[50085]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-09-17T07:24:22.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:22 smithi044 ceph-mon[58191]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 90/999 objects degraded (9.009%), 13 pgs degraded) 2024-09-17T07:24:22.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:22 smithi044 ceph-mon[60356]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 90/999 objects degraded (9.009%), 13 pgs degraded) 2024-09-17T07:24:22.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:22 smithi179 ceph-mon[50085]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 90/999 objects degraded (9.009%), 13 pgs degraded) 2024-09-17T07:24:23.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:23 smithi044 ceph-mon[58191]: pgmap v189: 161 pgs: 161 active+clean; 604 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 168 KiB/s rd, 0 B/s wr, 280 op/s; 0 B/s, 10 objects/s recovering 2024-09-17T07:24:23.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:23 smithi044 ceph-mon[60356]: pgmap v189: 161 pgs: 161 active+clean; 604 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 168 KiB/s rd, 0 B/s wr, 280 op/s; 0 B/s, 10 objects/s recovering 2024-09-17T07:24:23.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:23 smithi179 ceph-mon[50085]: pgmap v189: 161 pgs: 161 active+clean; 604 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 168 KiB/s rd, 0 B/s wr, 280 op/s; 0 B/s, 10 objects/s recovering 2024-09-17T07:24:25.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:25 smithi044 ceph-mon[58191]: pgmap v190: 161 pgs: 161 active+clean; 604 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 178 KiB/s rd, 0 B/s wr, 296 op/s; 0 B/s, 8 objects/s recovering 2024-09-17T07:24:25.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:25 smithi044 ceph-mon[60356]: pgmap v190: 161 pgs: 161 active+clean; 604 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 178 KiB/s rd, 0 B/s wr, 296 op/s; 0 B/s, 8 objects/s recovering 2024-09-17T07:24:25.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:25 smithi179 ceph-mon[50085]: pgmap v190: 161 pgs: 161 active+clean; 604 KiB data, 279 MiB used, 715 GiB / 715 GiB avail; 178 KiB/s rd, 0 B/s wr, 296 op/s; 0 B/s, 8 objects/s recovering 2024-09-17T07:24:27.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:27 smithi044 ceph-mon[58191]: pgmap v191: 161 pgs: 161 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail; 209 KiB/s rd, 0 B/s wr, 347 op/s; 0 B/s, 6 objects/s recovering 2024-09-17T07:24:27.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:27 smithi044 ceph-mon[60356]: pgmap v191: 161 pgs: 161 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail; 209 KiB/s rd, 0 B/s wr, 347 op/s; 0 B/s, 6 objects/s recovering 2024-09-17T07:24:27.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:27 smithi179 ceph-mon[50085]: pgmap v191: 161 pgs: 161 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail; 209 KiB/s rd, 0 B/s wr, 347 op/s; 0 B/s, 6 objects/s recovering 2024-09-17T07:24:29.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:29 smithi044 ceph-mon[60356]: pgmap v192: 161 pgs: 161 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail; 159 KiB/s rd, 0 B/s wr, 265 op/s; 0 B/s, 4 objects/s recovering 2024-09-17T07:24:29.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:29 smithi044 ceph-mon[58191]: pgmap v192: 161 pgs: 161 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail; 159 KiB/s rd, 0 B/s wr, 265 op/s; 0 B/s, 4 objects/s recovering 2024-09-17T07:24:29.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:29 smithi179 ceph-mon[50085]: pgmap v192: 161 pgs: 161 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail; 159 KiB/s rd, 0 B/s wr, 265 op/s; 0 B/s, 4 objects/s recovering 2024-09-17T07:24:31.574 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:31 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:31.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:31 smithi044 ceph-mon[60356]: pgmap v193: 161 pgs: 161 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail; 133 KiB/s rd, 0 B/s wr, 222 op/s; 0 B/s, 4 objects/s recovering 2024-09-17T07:24:31.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:31 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:31.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:31 smithi044 ceph-mon[58191]: pgmap v193: 161 pgs: 161 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail; 133 KiB/s rd, 0 B/s wr, 222 op/s; 0 B/s, 4 objects/s recovering 2024-09-17T07:24:31.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:31 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:31.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:31 smithi179 ceph-mon[50085]: pgmap v193: 161 pgs: 161 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail; 133 KiB/s rd, 0 B/s wr, 222 op/s; 0 B/s, 4 objects/s recovering 2024-09-17T07:24:33.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:33 smithi044 ceph-mon[60356]: pgmap v194: 161 pgs: 161 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 0 B/s wr, 185 op/s; 0 B/s, 3 objects/s recovering 2024-09-17T07:24:33.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:33 smithi044 ceph-mon[58191]: pgmap v194: 161 pgs: 161 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 0 B/s wr, 185 op/s; 0 B/s, 3 objects/s recovering 2024-09-17T07:24:33.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:33 smithi179 ceph-mon[50085]: pgmap v194: 161 pgs: 161 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 0 B/s wr, 185 op/s; 0 B/s, 3 objects/s recovering 2024-09-17T07:24:35.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:35 smithi044 ceph-mon[60356]: pgmap v195: 161 pgs: 161 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-17T07:24:35.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:35 smithi044 ceph-mon[58191]: pgmap v195: 161 pgs: 161 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-17T07:24:35.861 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:35 smithi179 ceph-mon[50085]: pgmap v195: 161 pgs: 161 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s 2024-09-17T07:24:36.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:36 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T07:24:36.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:36 smithi044 ceph-mon[58191]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T07:24:36.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:36 smithi044 ceph-mon[58191]: Upgrade: osd.7 is safe to restart 2024-09-17T07:24:36.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:36 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T07:24:36.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:36 smithi044 ceph-mon[60356]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T07:24:36.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:36 smithi044 ceph-mon[60356]: Upgrade: osd.7 is safe to restart 2024-09-17T07:24:36.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:36 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T07:24:36.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:36 smithi179 ceph-mon[50085]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T07:24:36.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:36 smithi179 ceph-mon[50085]: Upgrade: osd.7 is safe to restart 2024-09-17T07:24:37.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:37 smithi179 ceph-mon[50085]: pgmap v196: 161 pgs: 161 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 78 op/s 2024-09-17T07:24:37.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:37 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:37.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:37 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-17T07:24:37.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:37 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:24:37.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:37 smithi044 ceph-mon[58191]: pgmap v196: 161 pgs: 161 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 78 op/s 2024-09-17T07:24:37.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:37 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:37.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:37 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-17T07:24:37.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:37 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:24:37.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:37 smithi044 ceph-mon[60356]: pgmap v196: 161 pgs: 161 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 78 op/s 2024-09-17T07:24:37.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:37 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:37.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:37 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-17T07:24:37.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:37 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:24:38.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:38 smithi179 ceph-mon[50085]: Upgrade: Updating osd.7 2024-09-17T07:24:38.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:38 smithi179 ceph-mon[50085]: Deploying daemon osd.7 on smithi179 2024-09-17T07:24:38.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:38 smithi044 ceph-mon[58191]: Upgrade: Updating osd.7 2024-09-17T07:24:38.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:38 smithi044 ceph-mon[58191]: Deploying daemon osd.7 on smithi179 2024-09-17T07:24:38.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:38 smithi044 ceph-mon[60356]: Upgrade: Updating osd.7 2024-09-17T07:24:38.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:38 smithi044 ceph-mon[60356]: Deploying daemon osd.7 on smithi179 2024-09-17T07:24:39.565 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:39 smithi179 ceph-mon[50085]: pgmap v197: 161 pgs: 161 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-09-17T07:24:39.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:39 smithi044 ceph-mon[58191]: pgmap v197: 161 pgs: 161 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-09-17T07:24:39.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:39 smithi044 ceph-mon[60356]: pgmap v197: 161 pgs: 161 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-09-17T07:24:40.338 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:39 smithi179 systemd[1]: Stopping Ceph osd.7 for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:24:40.338 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:40 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.7[33776]: debug 2024-09-17T07:24:40.051+0000 7fa537677700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-17T07:24:40.338 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:40 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.7[33776]: debug 2024-09-17T07:24:40.051+0000 7fa537677700 -1 osd.7 111 *** Got signal Terminated *** 2024-09-17T07:24:40.338 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:40 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.7[33776]: debug 2024-09-17T07:24:40.051+0000 7fa537677700 -1 osd.7 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T07:24:40.338 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:40 smithi179 podman[60250]: 2024-09-17 07:24:40.084336828 +0000 UTC m=+0.137475339 container died b0d186fbf8a6e8d3057a19eca570934e2bab16e1e2941daec756f22a82d96629 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.7, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-16.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Dimitri Savineau ) 2024-09-17T07:24:40.614 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:40 smithi179 podman[60250]: 2024-09-17 07:24:40.444911272 +0000 UTC m=+0.498049783 container cleanup b0d186fbf8a6e8d3057a19eca570934e2bab16e1e2941daec756f22a82d96629 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.7, GIT_BRANCH=HEAD, org.label-schema.build-date=20201204, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Dimitri Savineau , GIT_CLEAN=True, ceph=True, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, org.label-schema.name=CentOS Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-16.2.0) 2024-09-17T07:24:40.614 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:40 smithi179 bash[60250]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.7 2024-09-17T07:24:40.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:40 smithi179 ceph-mon[50085]: osd.7 reported immediately failed by osd.1 2024-09-17T07:24:40.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:40 smithi179 ceph-mon[50085]: osd.7 failed (root=default,host=smithi179) (connection refused reported by osd.1) 2024-09-17T07:24:40.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:40 smithi179 ceph-mon[50085]: osd.7 reported immediately failed by osd.0 2024-09-17T07:24:40.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:40 smithi179 ceph-mon[50085]: osd.7 reported immediately failed by osd.0 2024-09-17T07:24:40.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:40 smithi179 ceph-mon[50085]: osd.7 reported immediately failed by osd.0 2024-09-17T07:24:40.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:40 smithi179 ceph-mon[50085]: osd.7 reported immediately failed by osd.2 2024-09-17T07:24:40.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:40 smithi179 ceph-mon[50085]: osd.7 reported immediately failed by osd.6 2024-09-17T07:24:40.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:40 smithi179 ceph-mon[50085]: osd.7 reported immediately failed by osd.1 2024-09-17T07:24:40.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:40 smithi179 ceph-mon[50085]: osd.7 reported immediately failed by osd.2 2024-09-17T07:24:40.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:40 smithi179 ceph-mon[50085]: osd.7 reported immediately failed by osd.4 2024-09-17T07:24:40.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:40 smithi179 ceph-mon[50085]: osd.7 reported immediately failed by osd.6 2024-09-17T07:24:40.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:40 smithi179 ceph-mon[50085]: osd.7 reported immediately failed by osd.1 2024-09-17T07:24:40.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:40 smithi179 ceph-mon[50085]: osd.7 reported immediately failed by osd.4 2024-09-17T07:24:40.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:40 smithi179 ceph-mon[50085]: osd.7 reported immediately failed by osd.5 2024-09-17T07:24:40.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:40 smithi179 ceph-mon[50085]: osd.7 reported immediately failed by osd.3 2024-09-17T07:24:40.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:40 smithi179 ceph-mon[50085]: osd.7 reported immediately failed by osd.5 2024-09-17T07:24:40.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:40 smithi179 ceph-mon[50085]: osd.7 reported immediately failed by osd.3 2024-09-17T07:24:40.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:40 smithi044 ceph-mon[60356]: osd.7 reported immediately failed by osd.1 2024-09-17T07:24:40.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:40 smithi044 ceph-mon[60356]: osd.7 failed (root=default,host=smithi179) (connection refused reported by osd.1) 2024-09-17T07:24:40.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:40 smithi044 ceph-mon[60356]: osd.7 reported immediately failed by osd.0 2024-09-17T07:24:40.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:40 smithi044 ceph-mon[60356]: osd.7 reported immediately failed by osd.0 2024-09-17T07:24:40.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:40 smithi044 ceph-mon[60356]: osd.7 reported immediately failed by osd.0 2024-09-17T07:24:40.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:40 smithi044 ceph-mon[60356]: osd.7 reported immediately failed by osd.2 2024-09-17T07:24:40.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:40 smithi044 ceph-mon[60356]: osd.7 reported immediately failed by osd.6 2024-09-17T07:24:40.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:40 smithi044 ceph-mon[60356]: osd.7 reported immediately failed by osd.1 2024-09-17T07:24:40.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:40 smithi044 ceph-mon[60356]: osd.7 reported immediately failed by osd.2 2024-09-17T07:24:40.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:40 smithi044 ceph-mon[60356]: osd.7 reported immediately failed by osd.4 2024-09-17T07:24:40.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:40 smithi044 ceph-mon[60356]: osd.7 reported immediately failed by osd.6 2024-09-17T07:24:40.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:40 smithi044 ceph-mon[60356]: osd.7 reported immediately failed by osd.1 2024-09-17T07:24:40.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:40 smithi044 ceph-mon[60356]: osd.7 reported immediately failed by osd.4 2024-09-17T07:24:40.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:40 smithi044 ceph-mon[60356]: osd.7 reported immediately failed by osd.5 2024-09-17T07:24:40.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:40 smithi044 ceph-mon[60356]: osd.7 reported immediately failed by osd.3 2024-09-17T07:24:40.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:40 smithi044 ceph-mon[60356]: osd.7 reported immediately failed by osd.5 2024-09-17T07:24:40.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:40 smithi044 ceph-mon[60356]: osd.7 reported immediately failed by osd.3 2024-09-17T07:24:40.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:40 smithi044 ceph-mon[58191]: osd.7 reported immediately failed by osd.1 2024-09-17T07:24:40.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:40 smithi044 ceph-mon[58191]: osd.7 failed (root=default,host=smithi179) (connection refused reported by osd.1) 2024-09-17T07:24:40.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:40 smithi044 ceph-mon[58191]: osd.7 reported immediately failed by osd.0 2024-09-17T07:24:40.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:40 smithi044 ceph-mon[58191]: osd.7 reported immediately failed by osd.0 2024-09-17T07:24:40.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:40 smithi044 ceph-mon[58191]: osd.7 reported immediately failed by osd.0 2024-09-17T07:24:40.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:40 smithi044 ceph-mon[58191]: osd.7 reported immediately failed by osd.2 2024-09-17T07:24:40.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:40 smithi044 ceph-mon[58191]: osd.7 reported immediately failed by osd.6 2024-09-17T07:24:40.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:40 smithi044 ceph-mon[58191]: osd.7 reported immediately failed by osd.1 2024-09-17T07:24:40.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:40 smithi044 ceph-mon[58191]: osd.7 reported immediately failed by osd.2 2024-09-17T07:24:40.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:40 smithi044 ceph-mon[58191]: osd.7 reported immediately failed by osd.4 2024-09-17T07:24:40.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:40 smithi044 ceph-mon[58191]: osd.7 reported immediately failed by osd.6 2024-09-17T07:24:40.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:40 smithi044 ceph-mon[58191]: osd.7 reported immediately failed by osd.1 2024-09-17T07:24:40.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:40 smithi044 ceph-mon[58191]: osd.7 reported immediately failed by osd.4 2024-09-17T07:24:40.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:40 smithi044 ceph-mon[58191]: osd.7 reported immediately failed by osd.5 2024-09-17T07:24:40.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:40 smithi044 ceph-mon[58191]: osd.7 reported immediately failed by osd.3 2024-09-17T07:24:40.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:40 smithi044 ceph-mon[58191]: osd.7 reported immediately failed by osd.5 2024-09-17T07:24:40.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:40 smithi044 ceph-mon[58191]: osd.7 reported immediately failed by osd.3 2024-09-17T07:24:41.014 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-09-17T07:24:41.114 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:40 smithi179 podman[60262]: 2024-09-17 07:24:40.730139333 +0000 UTC m=+0.640445554 container remove b0d186fbf8a6e8d3057a19eca570934e2bab16e1e2941daec756f22a82d96629 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd.7, ceph=True, maintainer=Dimitri Savineau , org.label-schema.build-date=20201204, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-16.2.0, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=HEAD) 2024-09-17T07:24:41.480 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:41 smithi179 podman[60367]: 2024-09-17 07:24:41.077372827 +0000 UTC m=+0.016019945 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:24:41.481 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:41 smithi179 podman[60367]: 2024-09-17 07:24:41.18968635 +0000 UTC m=+0.128333443 container create fa3952b296b544a4b0db61b16928a3a2b7a3bab6a1e8edbfea33975a9a2aaedd (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-17T07:24:41.481 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:41 smithi179 podman[60367]: 2024-09-17 07:24:41.400976215 +0000 UTC m=+0.339623329 container init fa3952b296b544a4b0db61b16928a3a2b7a3bab6a1e8edbfea33975a9a2aaedd (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7-deactivate, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-17T07:24:41.481 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:41 smithi179 podman[60367]: 2024-09-17 07:24:41.404346083 +0000 UTC m=+0.342993180 container start fa3952b296b544a4b0db61b16928a3a2b7a3bab6a1e8edbfea33975a9a2aaedd (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-17T07:24:41.688 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T07:24:41.688 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.a smithi044 *:9093,9094 running (6m) 107s ago 11m 18.9M - 0.20.0 0881eb8f169f 0473d4b023ad 2024-09-17T07:24:41.689 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.a smithi179 *:3000 running (5m) 25s ago 10m 33.3M - 6.7.4 557c83e11646 43ab0e03dcac 2024-09-17T07:24:41.689 INFO:teuthology.orchestra.run.smithi044.stdout:iscsi.foo.smithi044.elmtfk smithi044 unknown 107s ago 10m - - 2024-09-17T07:24:41.689 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.x smithi179 *:8443,8765 running (5m) 25s ago 15m 438M - 18.2.4-855-g19e2a2be 980a0bfcd684 66b52dc26022 2024-09-17T07:24:41.689 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.y smithi044 *:8443,8765 running (5m) 107s ago 16m 482M - 18.2.4-855-g19e2a2be 980a0bfcd684 8080dd318a1c 2024-09-17T07:24:41.689 INFO:teuthology.orchestra.run.smithi044.stdout:mon.a smithi044 running (4m) 107s ago 16m 48.7M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 c3293e8cdbba 2024-09-17T07:24:41.689 INFO:teuthology.orchestra.run.smithi044.stdout:mon.b smithi179 running (4m) 25s ago 15m 43.0M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 ade749759184 2024-09-17T07:24:41.689 INFO:teuthology.orchestra.run.smithi044.stdout:mon.c smithi044 running (4m) 107s ago 15m 42.0M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 02f85d31e0bd 2024-09-17T07:24:41.689 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.a smithi044 *:9100 running (6m) 107s ago 11m 17.1M - 0.18.1 e5a616e4b9cf 8a2cde5bd9ed 2024-09-17T07:24:41.689 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.b smithi179 *:9100 running (6m) 25s ago 11m 8929k - 0.18.1 e5a616e4b9cf 04c63b5cb524 2024-09-17T07:24:41.689 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (3m) 107s ago 14m 52.4M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 3ba47a4e2e95 2024-09-17T07:24:41.689 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (2m) 107s ago 14m 50.5M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 7655e43d9944 2024-09-17T07:24:41.689 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (2m) 107s ago 14m 49.0M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 43a48289eada 2024-09-17T07:24:41.690 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi044 running (115s) 107s ago 13m 11.5M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 b4323c928f39 2024-09-17T07:24:41.690 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi179 running (94s) 25s ago 13m 47.3M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 917adc8feb0b 2024-09-17T07:24:41.690 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi179 running (62s) 25s ago 13m 41.6M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 0258eb8ed0e7 2024-09-17T07:24:41.690 INFO:teuthology.orchestra.run.smithi044.stdout:osd.6 smithi179 running (29s) 25s ago 12m 12.0M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 22baeb2f62cb 2024-09-17T07:24:41.690 INFO:teuthology.orchestra.run.smithi044.stdout:osd.7 smithi179 running (12m) 25s ago 12m 58.7M 4096M 16.2.0 24ecd6d5f14c b0d186fbf8a6 2024-09-17T07:24:41.690 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.a smithi179 *:9095 error 25s ago 12m - - 2024-09-17T07:24:41.690 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi044.nhyhnh smithi044 *:8000 running (10m) 107s ago 10m 39.1M - 16.2.0 24ecd6d5f14c 9b455479614f 2024-09-17T07:24:41.690 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi179.tqlzyy smithi179 *:8000 running (10m) 25s ago 10m 42.2M - 16.2.0 24ecd6d5f14c c8d1693241ce 2024-09-17T07:24:41.690 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi044.vnufiz smithi044 *:80 running (10m) 107s ago 10m 39.2M - 16.2.0 24ecd6d5f14c 6eae2bff5ec8 2024-09-17T07:24:41.690 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi179.ecjhll smithi179 *:80 running (10m) 25s ago 10m 42.9M - 16.2.0 24ecd6d5f14c 253109ee0394 2024-09-17T07:24:41.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:41 smithi044 ceph-mon[58191]: pgmap v198: 161 pgs: 161 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-09-17T07:24:41.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:41 smithi044 ceph-mon[58191]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T07:24:41.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:41 smithi044 ceph-mon[58191]: Health check failed: all OSDs are running quincy or later but require_osd_release < quincy (OSD_UPGRADE_FINISHED) 2024-09-17T07:24:41.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:41 smithi044 ceph-mon[58191]: osdmap e112: 8 total, 7 up, 8 in 2024-09-17T07:24:41.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:41 smithi044 ceph-mon[60356]: pgmap v198: 161 pgs: 161 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-09-17T07:24:41.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:41 smithi044 ceph-mon[60356]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T07:24:41.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:41 smithi044 ceph-mon[60356]: Health check failed: all OSDs are running quincy or later but require_osd_release < quincy (OSD_UPGRADE_FINISHED) 2024-09-17T07:24:41.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:41 smithi044 ceph-mon[60356]: osdmap e112: 8 total, 7 up, 8 in 2024-09-17T07:24:41.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:41 smithi179 ceph-mon[50085]: pgmap v198: 161 pgs: 161 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-09-17T07:24:41.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:41 smithi179 ceph-mon[50085]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T07:24:41.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:41 smithi179 ceph-mon[50085]: Health check failed: all OSDs are running quincy or later but require_osd_release < quincy (OSD_UPGRADE_FINISHED) 2024-09-17T07:24:41.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:41 smithi179 ceph-mon[50085]: osdmap e112: 8 total, 7 up, 8 in 2024-09-17T07:24:41.865 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:41 smithi179 podman[60367]: 2024-09-17 07:24:41.485113399 +0000 UTC m=+0.423760497 container attach fa3952b296b544a4b0db61b16928a3a2b7a3bab6a1e8edbfea33975a9a2aaedd (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-17T07:24:41.865 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:41 smithi179 podman[60367]: 2024-09-17 07:24:41.607316797 +0000 UTC m=+0.545963899 container died fa3952b296b544a4b0db61b16928a3a2b7a3bab6a1e8edbfea33975a9a2aaedd (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-17T07:24:42.088 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:24:42.089 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-09-17T07:24:42.089 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3 2024-09-17T07:24:42.089 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:24:42.089 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-09-17T07:24:42.089 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T07:24:42.089 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:24:42.089 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-09-17T07:24:42.089 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 7 2024-09-17T07:24:42.089 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:24:42.090 INFO:teuthology.orchestra.run.smithi044.stdout: "rgw": { 2024-09-17T07:24:42.090 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2024-09-17T07:24:42.090 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:24:42.090 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-09-17T07:24:42.090 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4, 2024-09-17T07:24:42.090 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 12 2024-09-17T07:24:42.090 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-09-17T07:24:42.090 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:24:42.427 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:24:42.428 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9", 2024-09-17T07:24:42.428 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-09-17T07:24:42.428 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T07:24:42.428 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [ 2024-09-17T07:24:42.428 INFO:teuthology.orchestra.run.smithi044.stdout: "mon", 2024-09-17T07:24:42.428 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr" 2024-09-17T07:24:42.428 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-09-17T07:24:42.428 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "12/23 daemons upgraded", 2024-09-17T07:24:42.428 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "Currently upgrading osd daemons", 2024-09-17T07:24:42.428 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-09-17T07:24:42.428 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:24:42.483 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:42 smithi179 podman[60389]: 2024-09-17 07:24:42.15553576 +0000 UTC m=+0.542760890 container remove fa3952b296b544a4b0db61b16928a3a2b7a3bab6a1e8edbfea33975a9a2aaedd (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-17T07:24:42.483 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:42 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.7.service: Deactivated successfully. 2024-09-17T07:24:42.483 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:42 smithi179 systemd[1]: Stopped Ceph osd.7 for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:24:42.483 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:42 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.7.service: Consumed 1.107s CPU time. 2024-09-17T07:24:42.483 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:42 smithi179 systemd[1]: Starting Ceph osd.7 for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:24:42.733 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:42 smithi179 ceph-mon[50085]: from='client.44922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:24:42.733 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:42 smithi179 ceph-mon[50085]: from='client.44928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:24:42.734 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:42 smithi179 ceph-mon[50085]: osdmap e113: 8 total, 7 up, 8 in 2024-09-17T07:24:42.734 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:42 smithi179 ceph-mon[50085]: from='client.44934 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:24:42.734 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:42 smithi179 ceph-mon[50085]: from='client.44940 172.21.15.44:0/862221367' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:24:42.734 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:42 smithi179 podman[60487]: 2024-09-17 07:24:42.492089207 +0000 UTC m=+0.016477709 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:24:42.734 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:42 smithi179 podman[60487]: 2024-09-17 07:24:42.590009949 +0000 UTC m=+0.114398437 container create 09cfd80c37b671d24874382a9f6b2229ab825d93f619c7de0ff85d75c4b4cc77 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7-activate, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-17T07:24:42.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:42 smithi044 ceph-mon[58191]: from='client.44922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:24:42.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:42 smithi044 ceph-mon[58191]: from='client.44928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:24:42.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:42 smithi044 ceph-mon[58191]: osdmap e113: 8 total, 7 up, 8 in 2024-09-17T07:24:42.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:42 smithi044 ceph-mon[58191]: from='client.44934 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:24:42.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:42 smithi044 ceph-mon[58191]: from='client.44940 172.21.15.44:0/862221367' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:24:42.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:42 smithi044 ceph-mon[60356]: from='client.44922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:24:42.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:42 smithi044 ceph-mon[60356]: from='client.44928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:24:42.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:42 smithi044 ceph-mon[60356]: osdmap e113: 8 total, 7 up, 8 in 2024-09-17T07:24:42.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:42 smithi044 ceph-mon[60356]: from='client.44934 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:24:42.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:42 smithi044 ceph-mon[60356]: from='client.44940 172.21.15.44:0/862221367' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:24:42.826 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_WARN 2 failed cephadm daemon(s); 1 osds down; all OSDs are running quincy or later but require_osd_release < quincy; Reduced data availability: 3 pgs inactive, 5 pgs peering 2024-09-17T07:24:42.827 INFO:teuthology.orchestra.run.smithi044.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T07:24:42.828 INFO:teuthology.orchestra.run.smithi044.stdout: daemon iscsi.foo.smithi044.elmtfk on smithi044 is in unknown state 2024-09-17T07:24:42.828 INFO:teuthology.orchestra.run.smithi044.stdout: daemon prometheus.a on smithi179 is in error state 2024-09-17T07:24:42.828 INFO:teuthology.orchestra.run.smithi044.stdout:[WRN] OSD_DOWN: 1 osds down 2024-09-17T07:24:42.828 INFO:teuthology.orchestra.run.smithi044.stdout: osd.7 (root=default,host=smithi179) is down 2024-09-17T07:24:42.828 INFO:teuthology.orchestra.run.smithi044.stdout:[WRN] OSD_UPGRADE_FINISHED: all OSDs are running quincy or later but require_osd_release < quincy 2024-09-17T07:24:42.828 INFO:teuthology.orchestra.run.smithi044.stdout: all OSDs are running quincy or later but require_osd_release < quincy 2024-09-17T07:24:42.828 INFO:teuthology.orchestra.run.smithi044.stdout:[WRN] PG_AVAILABILITY: Reduced data availability: 3 pgs inactive, 5 pgs peering 2024-09-17T07:24:42.828 INFO:teuthology.orchestra.run.smithi044.stdout: pg 2.b is stuck peering for 11m, current state peering, last acting [4,5] 2024-09-17T07:24:42.828 INFO:teuthology.orchestra.run.smithi044.stdout: pg 5.4 is stuck peering for 65s, current state peering, last acting [2,5] 2024-09-17T07:24:42.828 INFO:teuthology.orchestra.run.smithi044.stdout: pg 5.1a is stuck peering for 91s, current state peering, last acting [4,1] 2024-09-17T07:24:42.828 INFO:teuthology.orchestra.run.smithi044.stdout: pg 6.8 is stuck peering for 111s, current state peering, last acting [2,3] 2024-09-17T07:24:42.828 INFO:teuthology.orchestra.run.smithi044.stdout: pg 6.12 is stuck peering for 91s, current state peering, last acting [2,4] 2024-09-17T07:24:43.114 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:42 smithi179 podman[60487]: 2024-09-17 07:24:42.818331086 +0000 UTC m=+0.342719582 container init 09cfd80c37b671d24874382a9f6b2229ab825d93f619c7de0ff85d75c4b4cc77 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7-activate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2) 2024-09-17T07:24:43.114 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:42 smithi179 podman[60487]: 2024-09-17 07:24:42.821775283 +0000 UTC m=+0.346163775 container start 09cfd80c37b671d24874382a9f6b2229ab825d93f619c7de0ff85d75c4b4cc77 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7-activate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-17T07:24:43.114 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:42 smithi179 podman[60487]: 2024-09-17 07:24:42.88496968 +0000 UTC m=+0.409358174 container attach 09cfd80c37b671d24874382a9f6b2229ab825d93f619c7de0ff85d75c4b4cc77 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-17T07:24:43.464 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:43 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7-activate[60502]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-17T07:24:43.465 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:43 smithi179 bash[60487]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-17T07:24:43.465 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:43 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7-activate[60502]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-17T07:24:43.465 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:43 smithi179 bash[60487]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-17T07:24:43.465 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:43 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7-activate[60502]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-17T07:24:43.465 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:43 smithi179 bash[60487]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-17T07:24:43.465 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:43 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7-activate[60502]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-17T07:24:43.465 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:43 smithi179 bash[60487]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-17T07:24:43.465 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:43 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7-activate[60502]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-09-17T07:24:43.465 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:43 smithi179 bash[60487]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-09-17T07:24:43.465 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:43 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7-activate[60502]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-17T07:24:43.465 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:43 smithi179 bash[60487]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-17T07:24:43.465 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:43 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7-activate[60502]: --> ceph-volume raw activate successful for osd ID: 7 2024-09-17T07:24:43.465 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:43 smithi179 bash[60487]: --> ceph-volume raw activate successful for osd ID: 7 2024-09-17T07:24:43.466 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:43 smithi179 podman[60487]: 2024-09-17 07:24:43.168548995 +0000 UTC m=+0.692937492 container died 09cfd80c37b671d24874382a9f6b2229ab825d93f619c7de0ff85d75c4b4cc77 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7-activate, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-17T07:24:43.789 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:43 smithi179 ceph-mon[50085]: pgmap v201: 161 pgs: 42 peering, 7 stale+active+clean, 112 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:24:43.789 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:43 smithi179 ceph-mon[50085]: from='client.54832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:24:43.789 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:43 smithi179 ceph-mon[50085]: Health check failed: Reduced data availability: 3 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-17T07:24:43.789 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:43 smithi179 ceph-mon[50085]: from='client.44952 172.21.15.44:0/2866627497' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:24:43.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:43 smithi044 ceph-mon[58191]: pgmap v201: 161 pgs: 42 peering, 7 stale+active+clean, 112 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:24:43.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:43 smithi044 ceph-mon[58191]: from='client.54832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:24:43.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:43 smithi044 ceph-mon[58191]: Health check failed: Reduced data availability: 3 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-17T07:24:43.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:43 smithi044 ceph-mon[58191]: from='client.44952 172.21.15.44:0/2866627497' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:24:43.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:43 smithi044 ceph-mon[60356]: pgmap v201: 161 pgs: 42 peering, 7 stale+active+clean, 112 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:24:43.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:43 smithi044 ceph-mon[60356]: from='client.54832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:24:43.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:43 smithi044 ceph-mon[60356]: Health check failed: Reduced data availability: 3 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-17T07:24:43.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:43 smithi044 ceph-mon[60356]: from='client.44952 172.21.15.44:0/2866627497' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:24:44.114 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:43 smithi179 podman[60624]: 2024-09-17 07:24:43.789328828 +0000 UTC m=+0.609745301 container remove 09cfd80c37b671d24874382a9f6b2229ab825d93f619c7de0ff85d75c4b4cc77 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-17T07:24:44.114 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:43 smithi179 podman[60685]: 2024-09-17 07:24:43.899649756 +0000 UTC m=+0.016557820 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:24:44.114 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:43 smithi179 podman[60685]: 2024-09-17 07:24:43.998978011 +0000 UTC m=+0.115886073 container create 36b85c18e11f02852d35fcf1e085d24df28704b4172eecdf944e718f28941bb3 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-17T07:24:44.614 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:44 smithi179 podman[60685]: 2024-09-17 07:24:44.260174018 +0000 UTC m=+0.377082087 container init 36b85c18e11f02852d35fcf1e085d24df28704b4172eecdf944e718f28941bb3 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-17T07:24:44.614 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:44 smithi179 podman[60685]: 2024-09-17 07:24:44.26349332 +0000 UTC m=+0.380401385 container start 36b85c18e11f02852d35fcf1e085d24df28704b4172eecdf944e718f28941bb3 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-17T07:24:44.614 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:44 smithi179 ceph-osd[60705]: -- 172.21.15.179:0/2347805860 <== mon.2 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5637a3f791e0 con 0x5637a4cf8400 2024-09-17T07:24:44.614 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:44 smithi179 bash[60685]: 36b85c18e11f02852d35fcf1e085d24df28704b4172eecdf944e718f28941bb3 2024-09-17T07:24:44.614 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:44 smithi179 systemd[1]: Started Ceph osd.7 for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:24:45.060 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:44 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:24:44.799+0000 7fd671c21640 -1 Falling back to public interface 2024-09-17T07:24:45.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:45 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:45.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:45 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:45.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:45 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:24:45.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:45 smithi179 ceph-mon[50085]: pgmap v202: 161 pgs: 7 active+undersized, 42 peering, 3 stale+active+clean, 7 active+undersized+degraded, 102 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail; 30/999 objects degraded (3.003%) 2024-09-17T07:24:45.627 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:45 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:45.627 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:45 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:45.628 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:45 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:24:45.628 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:45 smithi044 ceph-mon[58191]: pgmap v202: 161 pgs: 7 active+undersized, 42 peering, 3 stale+active+clean, 7 active+undersized+degraded, 102 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail; 30/999 objects degraded (3.003%) 2024-09-17T07:24:45.628 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:45 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:45.628 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:45 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:45.628 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:45 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:24:45.628 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:45 smithi044 ceph-mon[60356]: pgmap v202: 161 pgs: 7 active+undersized, 42 peering, 3 stale+active+clean, 7 active+undersized+degraded, 102 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail; 30/999 objects degraded (3.003%) 2024-09-17T07:24:46.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:46 smithi179 ceph-mon[50085]: Health check failed: Degraded data redundancy: 30/999 objects degraded (3.003%), 7 pgs degraded (PG_DEGRADED) 2024-09-17T07:24:46.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:46 smithi044 ceph-mon[58191]: Health check failed: Degraded data redundancy: 30/999 objects degraded (3.003%), 7 pgs degraded (PG_DEGRADED) 2024-09-17T07:24:46.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:46 smithi044 ceph-mon[60356]: Health check failed: Degraded data redundancy: 30/999 objects degraded (3.003%), 7 pgs degraded (PG_DEGRADED) 2024-09-17T07:24:47.580 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:47 smithi179 ceph-mon[50085]: pgmap v203: 161 pgs: 43 active+undersized, 26 active+undersized+degraded, 92 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail; 130/999 objects degraded (13.013%) 2024-09-17T07:24:47.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:47 smithi044 ceph-mon[58191]: pgmap v203: 161 pgs: 43 active+undersized, 26 active+undersized+degraded, 92 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail; 130/999 objects degraded (13.013%) 2024-09-17T07:24:47.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:47 smithi044 ceph-mon[60356]: pgmap v203: 161 pgs: 43 active+undersized, 26 active+undersized+degraded, 92 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail; 130/999 objects degraded (13.013%) 2024-09-17T07:24:47.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:24:47.579+0000 7fd671c21640 -1 osd.7 0 read_superblock omap replica is missing. 2024-09-17T07:24:47.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:47 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:24:47.613+0000 7fd671c21640 -1 osd.7 111 log_to_monitors true 2024-09-17T07:24:48.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:48 smithi044 ceph-mon[58191]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 5 pgs peering) 2024-09-17T07:24:48.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:48 smithi044 ceph-mon[58191]: from='osd.7 [v2:172.21.15.179:6824/2755074359,v1:172.21.15.179:6825/2755074359]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T07:24:48.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:48 smithi044 ceph-mon[58191]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T07:24:48.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:48 smithi044 ceph-mon[60356]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 5 pgs peering) 2024-09-17T07:24:48.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:48 smithi044 ceph-mon[60356]: from='osd.7 [v2:172.21.15.179:6824/2755074359,v1:172.21.15.179:6825/2755074359]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T07:24:48.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:48 smithi044 ceph-mon[60356]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T07:24:48.847 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:48 smithi179 ceph-mon[50085]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 5 pgs peering) 2024-09-17T07:24:48.847 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:48 smithi179 ceph-mon[50085]: from='osd.7 [v2:172.21.15.179:6824/2755074359,v1:172.21.15.179:6825/2755074359]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T07:24:48.847 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:48 smithi179 ceph-mon[50085]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T07:24:49.114 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:24:48 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:24:48.846+0000 7fd6659c3640 -1 osd.7 111 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T07:24:49.690 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:49 smithi179 ceph-mon[50085]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-17T07:24:49.690 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:49 smithi179 ceph-mon[50085]: osdmap e114: 8 total, 7 up, 8 in 2024-09-17T07:24:49.690 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:49 smithi179 ceph-mon[50085]: from='osd.7 [v2:172.21.15.179:6824/2755074359,v1:172.21.15.179:6825/2755074359]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:24:49.690 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:49 smithi179 ceph-mon[50085]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:24:49.690 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:49 smithi179 ceph-mon[50085]: pgmap v205: 161 pgs: 43 active+undersized, 26 active+undersized+degraded, 92 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail; 130/999 objects degraded (13.013%) 2024-09-17T07:24:49.690 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:49 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:49.690 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:49 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:49.690 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:49 smithi179 ceph-mon[50085]: from='osd.7 ' entity='osd.7' 2024-09-17T07:24:49.691 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:49 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:49.691 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:49 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:49.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:49 smithi044 ceph-mon[58191]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-17T07:24:49.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:49 smithi044 ceph-mon[58191]: osdmap e114: 8 total, 7 up, 8 in 2024-09-17T07:24:49.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:49 smithi044 ceph-mon[58191]: from='osd.7 [v2:172.21.15.179:6824/2755074359,v1:172.21.15.179:6825/2755074359]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:24:49.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:49 smithi044 ceph-mon[58191]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:24:49.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:49 smithi044 ceph-mon[58191]: pgmap v205: 161 pgs: 43 active+undersized, 26 active+undersized+degraded, 92 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail; 130/999 objects degraded (13.013%) 2024-09-17T07:24:49.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:49 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:49.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:49 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:49.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:49 smithi044 ceph-mon[58191]: from='osd.7 ' entity='osd.7' 2024-09-17T07:24:49.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:49 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:49.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:49 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:49.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:49 smithi044 ceph-mon[60356]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-17T07:24:49.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:49 smithi044 ceph-mon[60356]: osdmap e114: 8 total, 7 up, 8 in 2024-09-17T07:24:49.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:49 smithi044 ceph-mon[60356]: from='osd.7 [v2:172.21.15.179:6824/2755074359,v1:172.21.15.179:6825/2755074359]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:24:49.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:49 smithi044 ceph-mon[60356]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-09-17T07:24:49.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:49 smithi044 ceph-mon[60356]: pgmap v205: 161 pgs: 43 active+undersized, 26 active+undersized+degraded, 92 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail; 130/999 objects degraded (13.013%) 2024-09-17T07:24:49.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:49 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:49.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:49 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:49.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:49 smithi044 ceph-mon[60356]: from='osd.7 ' entity='osd.7' 2024-09-17T07:24:49.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:49 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:49.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:49 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:50.654 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:50 smithi179 ceph-mon[50085]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T07:24:50.654 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:50 smithi179 ceph-mon[50085]: osd.7 [v2:172.21.15.179:6824/2755074359,v1:172.21.15.179:6825/2755074359] boot 2024-09-17T07:24:50.654 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:50 smithi179 ceph-mon[50085]: osdmap e115: 8 total, 8 up, 8 in 2024-09-17T07:24:50.654 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:50 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:24:50.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:50 smithi044 ceph-mon[58191]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T07:24:50.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:50 smithi044 ceph-mon[58191]: osd.7 [v2:172.21.15.179:6824/2755074359,v1:172.21.15.179:6825/2755074359] boot 2024-09-17T07:24:50.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:50 smithi044 ceph-mon[58191]: osdmap e115: 8 total, 8 up, 8 in 2024-09-17T07:24:50.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:50 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:24:50.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:50 smithi044 ceph-mon[60356]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T07:24:50.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:50 smithi044 ceph-mon[60356]: osd.7 [v2:172.21.15.179:6824/2755074359,v1:172.21.15.179:6825/2755074359] boot 2024-09-17T07:24:50.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:50 smithi044 ceph-mon[60356]: osdmap e115: 8 total, 8 up, 8 in 2024-09-17T07:24:50.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:50 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T07:24:52.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:51 smithi179 ceph-mon[50085]: pgmap v207: 161 pgs: 10 peering, 37 active+undersized, 22 active+undersized+degraded, 92 active+clean; 604 KiB data, 302 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T07:24:52.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:51 smithi179 ceph-mon[50085]: osdmap e116: 8 total, 8 up, 8 in 2024-09-17T07:24:52.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:51 smithi179 ceph-mon[50085]: Health check update: Degraded data redundancy: 115/999 objects degraded (11.512%), 22 pgs degraded (PG_DEGRADED) 2024-09-17T07:24:52.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:51 smithi044 ceph-mon[58191]: pgmap v207: 161 pgs: 10 peering, 37 active+undersized, 22 active+undersized+degraded, 92 active+clean; 604 KiB data, 302 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T07:24:52.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:51 smithi044 ceph-mon[58191]: osdmap e116: 8 total, 8 up, 8 in 2024-09-17T07:24:52.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:51 smithi044 ceph-mon[58191]: Health check update: Degraded data redundancy: 115/999 objects degraded (11.512%), 22 pgs degraded (PG_DEGRADED) 2024-09-17T07:24:52.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:51 smithi044 ceph-mon[60356]: pgmap v207: 161 pgs: 10 peering, 37 active+undersized, 22 active+undersized+degraded, 92 active+clean; 604 KiB data, 302 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T07:24:52.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:51 smithi044 ceph-mon[60356]: osdmap e116: 8 total, 8 up, 8 in 2024-09-17T07:24:52.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:51 smithi044 ceph-mon[60356]: Health check update: Degraded data redundancy: 115/999 objects degraded (11.512%), 22 pgs degraded (PG_DEGRADED) 2024-09-17T07:24:53.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[58191]: pgmap v209: 161 pgs: 30 peering, 18 active+undersized, 15 active+undersized+degraded, 98 active+clean; 604 KiB data, 302 MiB used, 715 GiB / 715 GiB avail; 77/999 objects degraded (7.708%) 2024-09-17T07:24:53.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:53.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:53.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:24:53.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:24:53.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[58191]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:24:53.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[58191]: pgmap v210: 161 pgs: 30 peering, 18 active+undersized, 15 active+undersized+degraded, 98 active+clean; 604 KiB data, 302 MiB used, 715 GiB / 715 GiB avail; 77/999 objects degraded (7.708%) 2024-09-17T07:24:53.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[58191]: pgmap v211: 161 pgs: 30 peering, 18 active+undersized, 15 active+undersized+degraded, 98 active+clean; 604 KiB data, 302 MiB used, 715 GiB / 715 GiB avail; 77/999 objects degraded (7.708%) 2024-09-17T07:24:53.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:53.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:24:53.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:24:53.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:24:53.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:24:53.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:24:53.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[58191]: Upgrade: Setting container_image for all osd 2024-09-17T07:24:53.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:53.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-09-17T07:24:53.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-09-17T07:24:53.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-09-17T07:24:53.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-09-17T07:24:53.826 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-09-17T07:24:53.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-09-17T07:24:53.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-09-17T07:24:53.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-09-17T07:24:53.827 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-09-17T07:24:53.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[60356]: pgmap v209: 161 pgs: 30 peering, 18 active+undersized, 15 active+undersized+degraded, 98 active+clean; 604 KiB data, 302 MiB used, 715 GiB / 715 GiB avail; 77/999 objects degraded (7.708%) 2024-09-17T07:24:53.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:53.827 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:53.828 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:24:53.828 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:24:53.828 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[60356]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:24:53.828 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[60356]: pgmap v210: 161 pgs: 30 peering, 18 active+undersized, 15 active+undersized+degraded, 98 active+clean; 604 KiB data, 302 MiB used, 715 GiB / 715 GiB avail; 77/999 objects degraded (7.708%) 2024-09-17T07:24:53.828 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[60356]: pgmap v211: 161 pgs: 30 peering, 18 active+undersized, 15 active+undersized+degraded, 98 active+clean; 604 KiB data, 302 MiB used, 715 GiB / 715 GiB avail; 77/999 objects degraded (7.708%) 2024-09-17T07:24:53.828 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:53.828 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:24:53.828 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:24:53.828 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:24:53.828 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:24:53.828 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:24:53.828 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[60356]: Upgrade: Setting container_image for all osd 2024-09-17T07:24:53.829 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:53.829 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-09-17T07:24:53.829 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-09-17T07:24:53.829 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-09-17T07:24:53.829 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-09-17T07:24:53.829 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-09-17T07:24:53.829 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-09-17T07:24:53.829 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-09-17T07:24:53.829 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-09-17T07:24:53.829 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-09-17T07:24:53.829 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-09-17T07:24:53.829 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-09-17T07:24:53.829 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-09-17T07:24:53.829 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-09-17T07:24:53.830 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-09-17T07:24:53.830 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-09-17T07:24:53.830 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-09-17T07:24:53.830 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[60356]: Upgrade: Setting require_osd_release to 18 reef 2024-09-17T07:24:53.830 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "reef"}]: dispatch 2024-09-17T07:24:53.830 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-09-17T07:24:53.830 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-09-17T07:24:53.830 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-09-17T07:24:53.830 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-09-17T07:24:53.830 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-09-17T07:24:53.830 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-09-17T07:24:53.831 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-09-17T07:24:53.831 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[58191]: Upgrade: Setting require_osd_release to 18 reef 2024-09-17T07:24:53.831 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:53 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "reef"}]: dispatch 2024-09-17T07:24:53.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:53 smithi179 ceph-mon[50085]: pgmap v209: 161 pgs: 30 peering, 18 active+undersized, 15 active+undersized+degraded, 98 active+clean; 604 KiB data, 302 MiB used, 715 GiB / 715 GiB avail; 77/999 objects degraded (7.708%) 2024-09-17T07:24:53.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:53 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:53.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:53 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:53.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:53 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:24:53.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:53 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:24:53.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:53 smithi179 ceph-mon[50085]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:24:53.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:53 smithi179 ceph-mon[50085]: pgmap v210: 161 pgs: 30 peering, 18 active+undersized, 15 active+undersized+degraded, 98 active+clean; 604 KiB data, 302 MiB used, 715 GiB / 715 GiB avail; 77/999 objects degraded (7.708%) 2024-09-17T07:24:53.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:53 smithi179 ceph-mon[50085]: pgmap v211: 161 pgs: 30 peering, 18 active+undersized, 15 active+undersized+degraded, 98 active+clean; 604 KiB data, 302 MiB used, 715 GiB / 715 GiB avail; 77/999 objects degraded (7.708%) 2024-09-17T07:24:53.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:53 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:53.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:53 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:24:53.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:53 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:24:53.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:53 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:24:53.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:53 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:24:53.865 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:53 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:24:53.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:53 smithi179 ceph-mon[50085]: Upgrade: Setting container_image for all osd 2024-09-17T07:24:53.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:53 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:53.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:53 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-09-17T07:24:53.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:53 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-09-17T07:24:53.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:53 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-09-17T07:24:53.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:53 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-09-17T07:24:53.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:53 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-09-17T07:24:53.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:53 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-09-17T07:24:53.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:53 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-09-17T07:24:53.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:53 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-09-17T07:24:53.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:53 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-09-17T07:24:53.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:53 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-09-17T07:24:53.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:53 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-09-17T07:24:53.866 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:53 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-09-17T07:24:53.867 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:53 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-09-17T07:24:53.867 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:53 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-09-17T07:24:53.867 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:53 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-09-17T07:24:53.867 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:53 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-09-17T07:24:53.867 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:53 smithi179 ceph-mon[50085]: Upgrade: Setting require_osd_release to 18 reef 2024-09-17T07:24:53.867 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:53 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "reef"}]: dispatch 2024-09-17T07:24:54.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:54 smithi044 ceph-mon[58191]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running quincy or later but require_osd_release < quincy) 2024-09-17T07:24:54.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:54 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "reef"}]': finished 2024-09-17T07:24:54.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:54 smithi044 ceph-mon[58191]: osdmap e117: 8 total, 8 up, 8 in 2024-09-17T07:24:54.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:54 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:24:54.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:54 smithi044 ceph-mon[58191]: Upgrade: Setting container_image for all mds 2024-09-17T07:24:54.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:54 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:54.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:54 smithi044 ceph-mon[60356]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running quincy or later but require_osd_release < quincy) 2024-09-17T07:24:54.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:54 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "reef"}]': finished 2024-09-17T07:24:54.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:54 smithi044 ceph-mon[60356]: osdmap e117: 8 total, 8 up, 8 in 2024-09-17T07:24:54.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:54 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:24:54.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:54 smithi044 ceph-mon[60356]: Upgrade: Setting container_image for all mds 2024-09-17T07:24:54.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:54 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:54.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:54 smithi179 ceph-mon[50085]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running quincy or later but require_osd_release < quincy) 2024-09-17T07:24:54.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:54 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "reef"}]': finished 2024-09-17T07:24:54.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:54 smithi179 ceph-mon[50085]: osdmap e117: 8 total, 8 up, 8 in 2024-09-17T07:24:54.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:54 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:24:54.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:54 smithi179 ceph-mon[50085]: Upgrade: Setting container_image for all mds 2024-09-17T07:24:54.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:54 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:55.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:55 smithi044 ceph-mon[58191]: pgmap v213: 161 pgs: 30 peering, 131 active+clean; 604 KiB data, 306 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:24:55.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:55 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:55.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:55 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi044.nhyhnh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:24:55.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:55 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:24:55.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:55 smithi044 ceph-mon[60356]: pgmap v213: 161 pgs: 30 peering, 131 active+clean; 604 KiB data, 306 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:24:55.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:55 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:55.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:55 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi044.nhyhnh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:24:55.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:55 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:24:55.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:55 smithi179 ceph-mon[50085]: pgmap v213: 161 pgs: 30 peering, 131 active+clean; 604 KiB data, 306 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:24:55.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:55 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:24:55.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:55 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi044.nhyhnh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:24:55.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:55 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:24:56.682 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:56 smithi044 ceph-mon[60356]: Upgrade: Updating rgw.foo.smithi044.nhyhnh (1/4) 2024-09-17T07:24:56.682 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:56 smithi044 ceph-mon[60356]: Deploying daemon rgw.foo.smithi044.nhyhnh on smithi044 2024-09-17T07:24:56.682 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:56 smithi044 ceph-mon[60356]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 77/999 objects degraded (7.708%), 15 pgs degraded) 2024-09-17T07:24:56.683 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:56 smithi044 ceph-mon[58191]: Upgrade: Updating rgw.foo.smithi044.nhyhnh (1/4) 2024-09-17T07:24:56.683 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:56 smithi044 ceph-mon[58191]: Deploying daemon rgw.foo.smithi044.nhyhnh on smithi044 2024-09-17T07:24:56.683 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:56 smithi044 ceph-mon[58191]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 77/999 objects degraded (7.708%), 15 pgs degraded) 2024-09-17T07:24:56.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:56 smithi179 ceph-mon[50085]: Upgrade: Updating rgw.foo.smithi044.nhyhnh (1/4) 2024-09-17T07:24:56.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:56 smithi179 ceph-mon[50085]: Deploying daemon rgw.foo.smithi044.nhyhnh on smithi044 2024-09-17T07:24:56.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:56 smithi179 ceph-mon[50085]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 77/999 objects degraded (7.708%), 15 pgs degraded) 2024-09-17T07:24:57.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:57 smithi044 ceph-mon[58191]: pgmap v214: 161 pgs: 161 active+clean; 604 KiB data, 306 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:24:57.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:57 smithi044 ceph-mon[60356]: pgmap v214: 161 pgs: 161 active+clean; 604 KiB data, 306 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:24:57.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:57 smithi179 ceph-mon[50085]: pgmap v214: 161 pgs: 161 active+clean; 604 KiB data, 306 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:24:59.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:24:59 smithi044 ceph-mon[60356]: pgmap v215: 161 pgs: 161 active+clean; 604 KiB data, 306 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:24:59.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:24:59 smithi044 ceph-mon[58191]: pgmap v215: 161 pgs: 161 active+clean; 604 KiB data, 306 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:24:59.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:24:59 smithi179 ceph-mon[50085]: pgmap v215: 161 pgs: 161 active+clean; 604 KiB data, 306 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:25:00.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:00 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:00.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:00 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:01.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:00 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:01.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:00 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:01.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:00 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:01.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:00 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:01.703 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:01 smithi044 ceph-mon[58191]: pgmap v216: 161 pgs: 161 active+clean; 604 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 23 op/s 2024-09-17T07:25:01.704 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:01 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:01.704 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:01 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi044.vnufiz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:25:01.704 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:01 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:25:01.704 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:01 smithi044 ceph-mon[60356]: pgmap v216: 161 pgs: 161 active+clean; 604 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 23 op/s 2024-09-17T07:25:01.704 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:01 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:01.704 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:01 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi044.vnufiz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:25:01.704 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:01 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:25:01.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:01 smithi179 ceph-mon[50085]: pgmap v216: 161 pgs: 161 active+clean; 604 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 23 op/s 2024-09-17T07:25:01.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:01 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:01.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:01 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi044.vnufiz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:25:01.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:01 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:25:02.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:02 smithi044 ceph-mon[58191]: Upgrade: Updating rgw.smpl.smithi044.vnufiz (2/4) 2024-09-17T07:25:02.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:02 smithi044 ceph-mon[58191]: Deploying daemon rgw.smpl.smithi044.vnufiz on smithi044 2024-09-17T07:25:02.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:02 smithi044 ceph-mon[60356]: Upgrade: Updating rgw.smpl.smithi044.vnufiz (2/4) 2024-09-17T07:25:02.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:02 smithi044 ceph-mon[60356]: Deploying daemon rgw.smpl.smithi044.vnufiz on smithi044 2024-09-17T07:25:02.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:02 smithi179 ceph-mon[50085]: Upgrade: Updating rgw.smpl.smithi044.vnufiz (2/4) 2024-09-17T07:25:02.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:02 smithi179 ceph-mon[50085]: Deploying daemon rgw.smpl.smithi044.vnufiz on smithi044 2024-09-17T07:25:03.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:03 smithi179 ceph-mon[50085]: pgmap v217: 161 pgs: 161 active+clean; 604 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-17T07:25:04.030 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:03 smithi044 ceph-mon[58191]: pgmap v217: 161 pgs: 161 active+clean; 604 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-17T07:25:04.030 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:03 smithi044 ceph-mon[60356]: pgmap v217: 161 pgs: 161 active+clean; 604 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 65 op/s 2024-09-17T07:25:05.871 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:05 smithi179 ceph-mon[50085]: pgmap v218: 161 pgs: 161 active+clean; 604 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 189 B/s wr, 115 op/s 2024-09-17T07:25:05.871 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:05 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:05.872 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:05 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:05.872 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:05 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:06.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:05 smithi044 ceph-mon[58191]: pgmap v218: 161 pgs: 161 active+clean; 604 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 189 B/s wr, 115 op/s 2024-09-17T07:25:06.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:05 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:06.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:05 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:06.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:05 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:06.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:05 smithi044 ceph-mon[60356]: pgmap v218: 161 pgs: 161 active+clean; 604 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 189 B/s wr, 115 op/s 2024-09-17T07:25:06.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:05 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:06.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:05 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:06.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:05 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:07.758 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:07 smithi179 ceph-mon[50085]: pgmap v219: 161 pgs: 161 active+clean; 604 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 170 B/s wr, 133 op/s 2024-09-17T07:25:07.758 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:07 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:07.758 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:07 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi179.tqlzyy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:25:07.758 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:07 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:25:08.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:07 smithi044 ceph-mon[60356]: pgmap v219: 161 pgs: 161 active+clean; 604 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 170 B/s wr, 133 op/s 2024-09-17T07:25:08.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:07 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:08.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:07 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi179.tqlzyy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:25:08.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:07 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:25:08.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:07 smithi044 ceph-mon[58191]: pgmap v219: 161 pgs: 161 active+clean; 604 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 170 B/s wr, 133 op/s 2024-09-17T07:25:08.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:07 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:08.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:07 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi179.tqlzyy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:25:08.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:07 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:25:08.753 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:08 smithi179 ceph-mon[50085]: Upgrade: Updating rgw.foo.smithi179.tqlzyy (3/4) 2024-09-17T07:25:08.753 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:08 smithi179 ceph-mon[50085]: Deploying daemon rgw.foo.smithi179.tqlzyy on smithi179 2024-09-17T07:25:09.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:08 smithi044 ceph-mon[60356]: Upgrade: Updating rgw.foo.smithi179.tqlzyy (3/4) 2024-09-17T07:25:09.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:08 smithi044 ceph-mon[60356]: Deploying daemon rgw.foo.smithi179.tqlzyy on smithi179 2024-09-17T07:25:09.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:08 smithi044 ceph-mon[58191]: Upgrade: Updating rgw.foo.smithi179.tqlzyy (3/4) 2024-09-17T07:25:09.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:08 smithi044 ceph-mon[58191]: Deploying daemon rgw.foo.smithi179.tqlzyy on smithi179 2024-09-17T07:25:09.891 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:09 smithi179 ceph-mon[50085]: pgmap v220: 161 pgs: 161 active+clean; 604 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s rd, 170 B/s wr, 149 op/s 2024-09-17T07:25:10.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:09 smithi044 ceph-mon[60356]: pgmap v220: 161 pgs: 161 active+clean; 604 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s rd, 170 B/s wr, 149 op/s 2024-09-17T07:25:10.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:09 smithi044 ceph-mon[58191]: pgmap v220: 161 pgs: 161 active+clean; 604 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s rd, 170 B/s wr, 149 op/s 2024-09-17T07:25:12.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:11 smithi044 ceph-mon[58191]: pgmap v221: 161 pgs: 161 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 117 KiB/s rd, 255 B/s wr, 186 op/s 2024-09-17T07:25:12.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:11 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:12.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:11 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:12.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:11 smithi044 ceph-mon[60356]: pgmap v221: 161 pgs: 161 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 117 KiB/s rd, 255 B/s wr, 186 op/s 2024-09-17T07:25:12.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:11 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:12.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:11 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:12.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:11 smithi179 ceph-mon[50085]: pgmap v221: 161 pgs: 161 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 117 KiB/s rd, 255 B/s wr, 186 op/s 2024-09-17T07:25:12.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:11 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:12.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:11 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:13.190 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-09-17T07:25:13.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:13 smithi044 ceph-mon[58191]: Upgrade: Updating rgw.smpl.smithi179.ecjhll (4/4) 2024-09-17T07:25:13.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:13 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:13.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:13 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi179.ecjhll", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:25:13.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:13 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:25:13.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:13 smithi044 ceph-mon[58191]: Deploying daemon rgw.smpl.smithi179.ecjhll on smithi179 2024-09-17T07:25:13.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:13 smithi044 ceph-mon[58191]: pgmap v222: 161 pgs: 161 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 123 KiB/s rd, 255 B/s wr, 194 op/s 2024-09-17T07:25:13.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:13 smithi044 ceph-mon[60356]: Upgrade: Updating rgw.smpl.smithi179.ecjhll (4/4) 2024-09-17T07:25:13.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:13 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:13.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:13 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi179.ecjhll", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:25:13.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:13 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:25:13.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:13 smithi044 ceph-mon[60356]: Deploying daemon rgw.smpl.smithi179.ecjhll on smithi179 2024-09-17T07:25:13.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:13 smithi044 ceph-mon[60356]: pgmap v222: 161 pgs: 161 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 123 KiB/s rd, 255 B/s wr, 194 op/s 2024-09-17T07:25:13.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:13 smithi179 ceph-mon[50085]: Upgrade: Updating rgw.smpl.smithi179.ecjhll (4/4) 2024-09-17T07:25:13.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:13 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:13.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:13 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi179.ecjhll", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T07:25:13.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:13 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:25:13.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:13 smithi179 ceph-mon[50085]: Deploying daemon rgw.smpl.smithi179.ecjhll on smithi179 2024-09-17T07:25:13.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:13 smithi179 ceph-mon[50085]: pgmap v222: 161 pgs: 161 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 123 KiB/s rd, 255 B/s wr, 194 op/s 2024-09-17T07:25:13.855 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T07:25:13.855 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.a smithi044 *:9093,9094 running (6m) 2m ago 11m 18.9M - 0.20.0 0881eb8f169f 0473d4b023ad 2024-09-17T07:25:13.856 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.a smithi179 *:3000 running (6m) 25s ago 11m 33.5M - 6.7.4 557c83e11646 43ab0e03dcac 2024-09-17T07:25:13.856 INFO:teuthology.orchestra.run.smithi044.stdout:iscsi.foo.smithi044.elmtfk smithi044 unknown 2m ago 10m - - 2024-09-17T07:25:13.856 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.x smithi179 *:8443,8765 running (5m) 25s ago 15m 438M - 18.2.4-855-g19e2a2be 980a0bfcd684 66b52dc26022 2024-09-17T07:25:13.856 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.y smithi044 *:8443,8765 running (6m) 2m ago 17m 482M - 18.2.4-855-g19e2a2be 980a0bfcd684 8080dd318a1c 2024-09-17T07:25:13.856 INFO:teuthology.orchestra.run.smithi044.stdout:mon.a smithi044 running (5m) 2m ago 17m 48.7M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 c3293e8cdbba 2024-09-17T07:25:13.856 INFO:teuthology.orchestra.run.smithi044.stdout:mon.b smithi179 running (4m) 25s ago 15m 44.5M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 ade749759184 2024-09-17T07:25:13.856 INFO:teuthology.orchestra.run.smithi044.stdout:mon.c smithi044 running (5m) 2m ago 16m 42.0M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 02f85d31e0bd 2024-09-17T07:25:13.856 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.a smithi044 *:9100 running (7m) 2m ago 12m 17.1M - 0.18.1 e5a616e4b9cf 8a2cde5bd9ed 2024-09-17T07:25:13.856 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.b smithi179 *:9100 running (6m) 25s ago 12m 8929k - 0.18.1 e5a616e4b9cf 04c63b5cb524 2024-09-17T07:25:13.856 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (3m) 2m ago 15m 52.4M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 3ba47a4e2e95 2024-09-17T07:25:13.856 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (3m) 2m ago 15m 50.5M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 7655e43d9944 2024-09-17T07:25:13.856 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (2m) 2m ago 14m 49.0M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 43a48289eada 2024-09-17T07:25:13.857 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi044 running (2m) 2m ago 14m 11.5M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 b4323c928f39 2024-09-17T07:25:13.857 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi179 running (2m) 25s ago 14m 53.0M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 917adc8feb0b 2024-09-17T07:25:13.857 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi179 running (94s) 25s ago 13m 45.5M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 0258eb8ed0e7 2024-09-17T07:25:13.857 INFO:teuthology.orchestra.run.smithi044.stdout:osd.6 smithi179 running (62s) 25s ago 13m 46.0M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 22baeb2f62cb 2024-09-17T07:25:13.857 INFO:teuthology.orchestra.run.smithi044.stdout:osd.7 smithi179 running (29s) 25s ago 13m 12.1M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 36b85c18e11f 2024-09-17T07:25:13.857 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.a smithi179 *:9095 error 25s ago 12m - - 2024-09-17T07:25:13.857 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi044.nhyhnh smithi044 *:8000 starting - - - - 2024-09-17T07:25:13.857 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi179.tqlzyy smithi179 *:8000 starting - - - - 2024-09-17T07:25:13.857 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi044.vnufiz smithi044 *:80 starting - - - - 2024-09-17T07:25:13.857 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi179.ecjhll smithi179 *:80 running (10m) 25s ago 10m 43.4M - 16.2.0 24ecd6d5f14c 253109ee0394 2024-09-17T07:25:14.255 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:25:14.255 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-09-17T07:25:14.255 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3 2024-09-17T07:25:14.255 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:25:14.255 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-09-17T07:25:14.255 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T07:25:14.255 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:25:14.255 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-09-17T07:25:14.256 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 8 2024-09-17T07:25:14.256 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:25:14.256 INFO:teuthology.orchestra.run.smithi044.stdout: "rgw": { 2024-09-17T07:25:14.256 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 1, 2024-09-17T07:25:14.256 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3 2024-09-17T07:25:14.256 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:25:14.256 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-09-17T07:25:14.256 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 1, 2024-09-17T07:25:14.256 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 16 2024-09-17T07:25:14.256 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-09-17T07:25:14.256 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:25:14.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:14 smithi044 ceph-mon[58191]: from='client.45018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:25:14.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:14 smithi044 ceph-mon[58191]: from='client.54907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:25:14.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:14 smithi044 ceph-mon[58191]: from='client.45030 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:25:14.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:14 smithi044 ceph-mon[58191]: from='client.45036 172.21.15.44:0/655276707' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:14.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:14 smithi044 ceph-mon[60356]: from='client.45018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:25:14.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:14 smithi044 ceph-mon[60356]: from='client.54907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:25:14.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:14 smithi044 ceph-mon[60356]: from='client.45030 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:25:14.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:14 smithi044 ceph-mon[60356]: from='client.45036 172.21.15.44:0/655276707' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:14.597 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:25:14.597 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9", 2024-09-17T07:25:14.597 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-09-17T07:25:14.598 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T07:25:14.598 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [ 2024-09-17T07:25:14.598 INFO:teuthology.orchestra.run.smithi044.stdout: "osd", 2024-09-17T07:25:14.598 INFO:teuthology.orchestra.run.smithi044.stdout: "mon", 2024-09-17T07:25:14.598 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr" 2024-09-17T07:25:14.598 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-09-17T07:25:14.598 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "13/23 daemons upgraded", 2024-09-17T07:25:14.598 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "Currently upgrading rgw daemons", 2024-09-17T07:25:14.598 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-09-17T07:25:14.598 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:25:14.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:14 smithi179 ceph-mon[50085]: from='client.45018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:25:14.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:14 smithi179 ceph-mon[50085]: from='client.54907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:25:14.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:14 smithi179 ceph-mon[50085]: from='client.45030 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:25:14.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:14 smithi179 ceph-mon[50085]: from='client.45036 172.21.15.44:0/655276707' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:15.003 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_WARN 2 failed cephadm daemon(s) 2024-09-17T07:25:15.003 INFO:teuthology.orchestra.run.smithi044.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T07:25:15.003 INFO:teuthology.orchestra.run.smithi044.stdout: daemon iscsi.foo.smithi044.elmtfk on smithi044 is in unknown state 2024-09-17T07:25:15.003 INFO:teuthology.orchestra.run.smithi044.stdout: daemon prometheus.a on smithi179 is in error state 2024-09-17T07:25:15.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:15 smithi044 ceph-mon[58191]: pgmap v223: 161 pgs: 161 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 126 KiB/s rd, 341 B/s wr, 197 op/s 2024-09-17T07:25:15.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:15 smithi044 ceph-mon[58191]: from='client.54925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:25:15.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:15 smithi044 ceph-mon[58191]: from='client.54931 172.21.15.44:0/1324772384' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:25:15.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:15 smithi044 ceph-mon[60356]: pgmap v223: 161 pgs: 161 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 126 KiB/s rd, 341 B/s wr, 197 op/s 2024-09-17T07:25:15.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:15 smithi044 ceph-mon[60356]: from='client.54925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:25:15.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:15 smithi044 ceph-mon[60356]: from='client.54931 172.21.15.44:0/1324772384' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:25:15.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:15 smithi179 ceph-mon[50085]: pgmap v223: 161 pgs: 161 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 126 KiB/s rd, 341 B/s wr, 197 op/s 2024-09-17T07:25:15.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:15 smithi179 ceph-mon[50085]: from='client.54925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:25:15.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:15 smithi179 ceph-mon[50085]: from='client.54931 172.21.15.44:0/1324772384' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:25:17.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:17 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:17.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:17 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:17.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:17 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:25:17.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:17 smithi179 ceph-mon[50085]: pgmap v224: 161 pgs: 161 active+clean; 604 KiB data, 315 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 169 op/s 2024-09-17T07:25:17.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:17 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:17.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:17 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:17.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:17 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:25:17.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:17 smithi044 ceph-mon[58191]: pgmap v224: 161 pgs: 161 active+clean; 604 KiB data, 315 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 169 op/s 2024-09-17T07:25:17.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:17 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:17.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:17 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:17.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:17 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:25:17.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:17 smithi044 ceph-mon[60356]: pgmap v224: 161 pgs: 161 active+clean; 604 KiB data, 315 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 169 op/s 2024-09-17T07:25:19.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:19 smithi179 ceph-mon[50085]: pgmap v225: 161 pgs: 161 active+clean; 604 KiB data, 315 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 170 B/s wr, 155 op/s 2024-09-17T07:25:19.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:19 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:19.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:19 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:19.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:19 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:19.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:19 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:20.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:19 smithi044 ceph-mon[58191]: pgmap v225: 161 pgs: 161 active+clean; 604 KiB data, 315 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 170 B/s wr, 155 op/s 2024-09-17T07:25:20.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:19 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:20.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:19 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:20.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:19 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:20.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:19 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:20.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:19 smithi044 ceph-mon[60356]: pgmap v225: 161 pgs: 161 active+clean; 604 KiB data, 315 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 170 B/s wr, 155 op/s 2024-09-17T07:25:20.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:19 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:20.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:19 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:20.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:19 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:20.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:19 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:21.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:20 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:21.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:20 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:21.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:20 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:21.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:20 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:21.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:20 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:21.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:20 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:21.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:20 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:21.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:20 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:21.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:20 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:21.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:20 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:21.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:20 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:21.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:20 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:22.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:21 smithi044 ceph-mon[58191]: pgmap v226: 161 pgs: 161 active+clean; 604 KiB data, 319 MiB used, 715 GiB / 715 GiB avail; 114 KiB/s rd, 255 B/s wr, 177 op/s 2024-09-17T07:25:22.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:21 smithi044 ceph-mon[60356]: pgmap v226: 161 pgs: 161 active+clean; 604 KiB data, 319 MiB used, 715 GiB / 715 GiB avail; 114 KiB/s rd, 255 B/s wr, 177 op/s 2024-09-17T07:25:22.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:21 smithi179 ceph-mon[50085]: pgmap v226: 161 pgs: 161 active+clean; 604 KiB data, 319 MiB used, 715 GiB / 715 GiB avail; 114 KiB/s rd, 255 B/s wr, 177 op/s 2024-09-17T07:25:24.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[60356]: pgmap v227: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 170 B/s wr, 165 op/s 2024-09-17T07:25:24.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[60356]: Detected new or changed devices on smithi179 2024-09-17T07:25:24.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:24.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:24.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:24.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:24.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:25:24.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:25:24.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:24.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:25:24.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:24.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:24.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:24.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:24.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:24.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:24.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:24.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi044.nhyhnh"}]: dispatch 2024-09-17T07:25:24.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi044.nhyhnh"}]': finished 2024-09-17T07:25:24.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi179.tqlzyy"}]: dispatch 2024-09-17T07:25:24.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi179.tqlzyy"}]': finished 2024-09-17T07:25:24.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi044.vnufiz"}]: dispatch 2024-09-17T07:25:24.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi044.vnufiz"}]': finished 2024-09-17T07:25:24.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi179.ecjhll"}]: dispatch 2024-09-17T07:25:24.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi179.ecjhll"}]': finished 2024-09-17T07:25:24.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:24.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:24.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:24.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:24.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:24.078 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[58191]: pgmap v227: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 170 B/s wr, 165 op/s 2024-09-17T07:25:24.078 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[58191]: Detected new or changed devices on smithi179 2024-09-17T07:25:24.078 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:24.078 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:24.078 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:24.078 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:24.078 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:25:24.078 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:25:24.078 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:24.078 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:25:24.078 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:24.078 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:24.079 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:24.079 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:24.079 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:24.079 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:24.079 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:24.079 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi044.nhyhnh"}]: dispatch 2024-09-17T07:25:24.079 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi044.nhyhnh"}]': finished 2024-09-17T07:25:24.079 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi179.tqlzyy"}]: dispatch 2024-09-17T07:25:24.079 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi179.tqlzyy"}]': finished 2024-09-17T07:25:24.079 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi044.vnufiz"}]: dispatch 2024-09-17T07:25:24.079 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi044.vnufiz"}]': finished 2024-09-17T07:25:24.079 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi179.ecjhll"}]: dispatch 2024-09-17T07:25:24.079 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi179.ecjhll"}]': finished 2024-09-17T07:25:24.079 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:24.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:24.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:24.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:24.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:23 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:24.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:23 smithi179 ceph-mon[50085]: pgmap v227: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 170 B/s wr, 165 op/s 2024-09-17T07:25:24.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:23 smithi179 ceph-mon[50085]: Detected new or changed devices on smithi179 2024-09-17T07:25:24.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:23 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:24.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:23 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:24.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:23 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:24.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:23 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:24.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:23 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:25:24.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:23 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:25:24.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:23 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:24.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:23 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:25:24.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:23 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:24.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:23 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:24.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:23 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:24.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:23 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:24.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:23 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:24.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:23 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:24.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:23 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:24.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:23 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi044.nhyhnh"}]: dispatch 2024-09-17T07:25:24.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:23 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi044.nhyhnh"}]': finished 2024-09-17T07:25:24.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:23 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi179.tqlzyy"}]: dispatch 2024-09-17T07:25:24.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:23 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi179.tqlzyy"}]': finished 2024-09-17T07:25:24.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:23 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi044.vnufiz"}]: dispatch 2024-09-17T07:25:24.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:23 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi044.vnufiz"}]': finished 2024-09-17T07:25:24.367 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:23 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi179.ecjhll"}]: dispatch 2024-09-17T07:25:24.367 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:23 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi179.ecjhll"}]': finished 2024-09-17T07:25:24.367 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:23 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:24.367 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:23 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:24.367 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:23 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:24.367 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:23 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:24.367 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:23 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:25.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:24 smithi044 ceph-mon[58191]: Detected new or changed devices on smithi044 2024-09-17T07:25:25.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:24 smithi044 ceph-mon[58191]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:25:25.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:24 smithi044 ceph-mon[58191]: pgmap v228: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s rd, 190 B/s wr, 158 op/s 2024-09-17T07:25:25.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:24 smithi044 ceph-mon[58191]: pgmap v229: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 116 B/s wr, 138 op/s 2024-09-17T07:25:25.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:24 smithi044 ceph-mon[58191]: Upgrade: Setting container_image for all rgw 2024-09-17T07:25:25.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:24 smithi044 ceph-mon[58191]: Upgrade: Setting container_image for all rbd-mirror 2024-09-17T07:25:25.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:24 smithi044 ceph-mon[58191]: Upgrade: Setting container_image for all ceph-exporter 2024-09-17T07:25:25.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:24 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:25.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:24 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi044.elmtfk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T07:25:25.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:24 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:25:25.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:24 smithi044 ceph-mon[60356]: Detected new or changed devices on smithi044 2024-09-17T07:25:25.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:24 smithi044 ceph-mon[60356]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:25:25.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:24 smithi044 ceph-mon[60356]: pgmap v228: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s rd, 190 B/s wr, 158 op/s 2024-09-17T07:25:25.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:24 smithi044 ceph-mon[60356]: pgmap v229: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 116 B/s wr, 138 op/s 2024-09-17T07:25:25.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:24 smithi044 ceph-mon[60356]: Upgrade: Setting container_image for all rgw 2024-09-17T07:25:25.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:24 smithi044 ceph-mon[60356]: Upgrade: Setting container_image for all rbd-mirror 2024-09-17T07:25:25.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:24 smithi044 ceph-mon[60356]: Upgrade: Setting container_image for all ceph-exporter 2024-09-17T07:25:25.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:24 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:25.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:24 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi044.elmtfk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T07:25:25.077 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:24 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:25:25.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:24 smithi179 ceph-mon[50085]: Detected new or changed devices on smithi044 2024-09-17T07:25:25.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:24 smithi179 ceph-mon[50085]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:25:25.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:24 smithi179 ceph-mon[50085]: pgmap v228: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s rd, 190 B/s wr, 158 op/s 2024-09-17T07:25:25.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:24 smithi179 ceph-mon[50085]: pgmap v229: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 116 B/s wr, 138 op/s 2024-09-17T07:25:25.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:24 smithi179 ceph-mon[50085]: Upgrade: Setting container_image for all rgw 2024-09-17T07:25:25.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:24 smithi179 ceph-mon[50085]: Upgrade: Setting container_image for all rbd-mirror 2024-09-17T07:25:25.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:24 smithi179 ceph-mon[50085]: Upgrade: Setting container_image for all ceph-exporter 2024-09-17T07:25:25.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:24 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:25.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:24 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi044.elmtfk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T07:25:25.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:24 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:25:26.232 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:25 smithi044 ceph-mon[60356]: Upgrade: Updating iscsi.foo.smithi044.elmtfk 2024-09-17T07:25:26.232 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:25 smithi044 ceph-mon[60356]: Deploying daemon iscsi.foo.smithi044.elmtfk on smithi044 2024-09-17T07:25:26.232 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:25 smithi044 ceph-mon[58191]: Upgrade: Updating iscsi.foo.smithi044.elmtfk 2024-09-17T07:25:26.232 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:25 smithi044 ceph-mon[58191]: Deploying daemon iscsi.foo.smithi044.elmtfk on smithi044 2024-09-17T07:25:26.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:25 smithi179 ceph-mon[50085]: Upgrade: Updating iscsi.foo.smithi044.elmtfk 2024-09-17T07:25:26.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:25 smithi179 ceph-mon[50085]: Deploying daemon iscsi.foo.smithi044.elmtfk on smithi044 2024-09-17T07:25:27.187 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:26 smithi044 ceph-mon[58191]: pgmap v230: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 116 B/s wr, 108 op/s 2024-09-17T07:25:27.187 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:26 smithi044 ceph-mon[60356]: pgmap v230: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 116 B/s wr, 108 op/s 2024-09-17T07:25:27.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:26 smithi179 ceph-mon[50085]: pgmap v230: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 116 B/s wr, 108 op/s 2024-09-17T07:25:29.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:28 smithi044 ceph-mon[58191]: pgmap v231: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 116 B/s wr, 86 op/s 2024-09-17T07:25:29.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:28 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:29.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:28 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:29.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:28 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:25:29.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:28 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/3777091404' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T07:25:29.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:28 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/1606394124' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: dispatch 2024-09-17T07:25:29.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:28 smithi044 ceph-mon[58191]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: dispatch 2024-09-17T07:25:29.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:28 smithi044 ceph-mon[60356]: pgmap v231: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 116 B/s wr, 86 op/s 2024-09-17T07:25:29.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:28 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:29.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:28 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:29.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:28 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:25:29.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:28 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/3777091404' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T07:25:29.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:28 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/1606394124' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: dispatch 2024-09-17T07:25:29.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:28 smithi044 ceph-mon[60356]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: dispatch 2024-09-17T07:25:29.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:28 smithi179 ceph-mon[50085]: pgmap v231: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 116 B/s wr, 86 op/s 2024-09-17T07:25:29.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:28 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:29.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:28 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:29.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:28 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:25:29.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:28 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/3777091404' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T07:25:29.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:28 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/1606394124' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: dispatch 2024-09-17T07:25:29.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:28 smithi179 ceph-mon[50085]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]: dispatch 2024-09-17T07:25:30.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:30 smithi044 ceph-mon[58191]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]': finished 2024-09-17T07:25:30.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:30 smithi044 ceph-mon[58191]: osdmap e118: 8 total, 8 up, 8 in 2024-09-17T07:25:30.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:30 smithi044 ceph-mon[58191]: pgmap v233: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:25:30.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:30 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/2549301269' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/1355368272"}]: dispatch 2024-09-17T07:25:30.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:30 smithi044 ceph-mon[60356]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]': finished 2024-09-17T07:25:30.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:30 smithi044 ceph-mon[60356]: osdmap e118: 8 total, 8 up, 8 in 2024-09-17T07:25:30.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:30 smithi044 ceph-mon[60356]: pgmap v233: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:25:30.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:30 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/2549301269' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/1355368272"}]: dispatch 2024-09-17T07:25:30.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:30 smithi179 ceph-mon[50085]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/2680338908"}]': finished 2024-09-17T07:25:30.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:30 smithi179 ceph-mon[50085]: osdmap e118: 8 total, 8 up, 8 in 2024-09-17T07:25:30.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:30 smithi179 ceph-mon[50085]: pgmap v233: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:25:30.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:30 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/2549301269' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/1355368272"}]: dispatch 2024-09-17T07:25:31.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:31 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/2549301269' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/1355368272"}]': finished 2024-09-17T07:25:31.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:31 smithi044 ceph-mon[58191]: osdmap e119: 8 total, 8 up, 8 in 2024-09-17T07:25:31.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:31 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/1674918929' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/2084457509"}]: dispatch 2024-09-17T07:25:31.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:31 smithi044 ceph-mon[58191]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/2084457509"}]: dispatch 2024-09-17T07:25:31.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:31 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/2549301269' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/1355368272"}]': finished 2024-09-17T07:25:31.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:31 smithi044 ceph-mon[60356]: osdmap e119: 8 total, 8 up, 8 in 2024-09-17T07:25:31.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:31 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/1674918929' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/2084457509"}]: dispatch 2024-09-17T07:25:31.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:31 smithi044 ceph-mon[60356]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/2084457509"}]: dispatch 2024-09-17T07:25:31.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:31 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/2549301269' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/1355368272"}]': finished 2024-09-17T07:25:31.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:31 smithi179 ceph-mon[50085]: osdmap e119: 8 total, 8 up, 8 in 2024-09-17T07:25:31.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:31 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/1674918929' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/2084457509"}]: dispatch 2024-09-17T07:25:31.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:31 smithi179 ceph-mon[50085]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/2084457509"}]: dispatch 2024-09-17T07:25:32.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:32 smithi044 ceph-mon[58191]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/2084457509"}]': finished 2024-09-17T07:25:32.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:32 smithi044 ceph-mon[58191]: osdmap e120: 8 total, 8 up, 8 in 2024-09-17T07:25:32.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:32 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:32.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:32 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:32.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:32 smithi044 ceph-mon[58191]: pgmap v236: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-17T07:25:32.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:32 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/695062658' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6801/3191260693"}]: dispatch 2024-09-17T07:25:32.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:32 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:32.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:32 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:32.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:32 smithi044 ceph-mon[60356]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/2084457509"}]': finished 2024-09-17T07:25:32.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:32 smithi044 ceph-mon[60356]: osdmap e120: 8 total, 8 up, 8 in 2024-09-17T07:25:32.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:32 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:32.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:32 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:32.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:32 smithi044 ceph-mon[60356]: pgmap v236: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-17T07:25:32.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:32 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/695062658' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6801/3191260693"}]: dispatch 2024-09-17T07:25:32.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:32 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:32.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:32 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:32.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:32 smithi179 ceph-mon[50085]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/2084457509"}]': finished 2024-09-17T07:25:32.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:32 smithi179 ceph-mon[50085]: osdmap e120: 8 total, 8 up, 8 in 2024-09-17T07:25:32.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:32 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:32.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:32 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:32.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:32 smithi179 ceph-mon[50085]: pgmap v236: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-17T07:25:32.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:32 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/695062658' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6801/3191260693"}]: dispatch 2024-09-17T07:25:32.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:32 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:32.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:32 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:33.159 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:33 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/695062658' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6801/3191260693"}]': finished 2024-09-17T07:25:33.159 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:33 smithi044 ceph-mon[58191]: osdmap e121: 8 total, 8 up, 8 in 2024-09-17T07:25:33.160 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:33 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/1964791057' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/155880459"}]: dispatch 2024-09-17T07:25:33.160 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:33 smithi044 ceph-mon[58191]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/155880459"}]: dispatch 2024-09-17T07:25:33.160 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:33 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:33.160 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:33 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:33.160 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:33 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/695062658' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6801/3191260693"}]': finished 2024-09-17T07:25:33.160 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:33 smithi044 ceph-mon[60356]: osdmap e121: 8 total, 8 up, 8 in 2024-09-17T07:25:33.160 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:33 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/1964791057' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/155880459"}]: dispatch 2024-09-17T07:25:33.160 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:33 smithi044 ceph-mon[60356]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/155880459"}]: dispatch 2024-09-17T07:25:33.161 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:33 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:33.161 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:33 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:33.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:33 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/695062658' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6801/3191260693"}]': finished 2024-09-17T07:25:33.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:33 smithi179 ceph-mon[50085]: osdmap e121: 8 total, 8 up, 8 in 2024-09-17T07:25:33.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:33 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/1964791057' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/155880459"}]: dispatch 2024-09-17T07:25:33.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:33 smithi179 ceph-mon[50085]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/155880459"}]: dispatch 2024-09-17T07:25:33.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:33 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:33.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:33 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:34.317 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:34 smithi044 ceph-mon[58191]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/155880459"}]': finished 2024-09-17T07:25:34.317 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:34 smithi044 ceph-mon[58191]: osdmap e122: 8 total, 8 up, 8 in 2024-09-17T07:25:34.317 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:34 smithi044 ceph-mon[58191]: pgmap v239: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-17T07:25:34.317 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:34 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/1138824611' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6800/2448217567"}]: dispatch 2024-09-17T07:25:34.317 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:34 smithi044 ceph-mon[60356]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/155880459"}]': finished 2024-09-17T07:25:34.318 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:34 smithi044 ceph-mon[60356]: osdmap e122: 8 total, 8 up, 8 in 2024-09-17T07:25:34.318 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:34 smithi044 ceph-mon[60356]: pgmap v239: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-17T07:25:34.318 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:34 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/1138824611' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6800/2448217567"}]: dispatch 2024-09-17T07:25:34.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:34 smithi179 ceph-mon[50085]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/155880459"}]': finished 2024-09-17T07:25:34.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:34 smithi179 ceph-mon[50085]: osdmap e122: 8 total, 8 up, 8 in 2024-09-17T07:25:34.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:34 smithi179 ceph-mon[50085]: pgmap v239: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-17T07:25:34.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:34 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/1138824611' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6800/2448217567"}]: dispatch 2024-09-17T07:25:35.478 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:35 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/1138824611' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6800/2448217567"}]': finished 2024-09-17T07:25:35.478 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:35 smithi044 ceph-mon[58191]: osdmap e123: 8 total, 8 up, 8 in 2024-09-17T07:25:35.478 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:35 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/4258601073' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/300796373"}]: dispatch 2024-09-17T07:25:35.478 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:35 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/1138824611' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6800/2448217567"}]': finished 2024-09-17T07:25:35.478 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:35 smithi044 ceph-mon[60356]: osdmap e123: 8 total, 8 up, 8 in 2024-09-17T07:25:35.479 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:35 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/4258601073' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/300796373"}]: dispatch 2024-09-17T07:25:35.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:35 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/1138824611' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6800/2448217567"}]': finished 2024-09-17T07:25:35.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:35 smithi179 ceph-mon[50085]: osdmap e123: 8 total, 8 up, 8 in 2024-09-17T07:25:35.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:35 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/4258601073' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/300796373"}]: dispatch 2024-09-17T07:25:36.515 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:36 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/4258601073' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/300796373"}]': finished 2024-09-17T07:25:36.515 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:36 smithi179 ceph-mon[50085]: osdmap e124: 8 total, 8 up, 8 in 2024-09-17T07:25:36.515 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:36 smithi179 ceph-mon[50085]: pgmap v242: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:25:36.516 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:36 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/2669037623' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6801/2448217567"}]: dispatch 2024-09-17T07:25:36.516 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:36 smithi179 ceph-mon[50085]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6801/2448217567"}]: dispatch 2024-09-17T07:25:36.516 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:36 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:36.516 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:36 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:36.516 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:36 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:25:36.516 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:36 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:25:36.516 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:36 smithi179 ceph-mon[50085]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:25:36.516 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:36 smithi179 ceph-mon[50085]: pgmap v243: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-17T07:25:36.516 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:36 smithi179 ceph-mon[50085]: pgmap v244: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-09-17T07:25:36.516 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:36 smithi179 ceph-mon[50085]: Checking dashboard <-> RGW credentials 2024-09-17T07:25:36.530 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:36 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/4258601073' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/300796373"}]': finished 2024-09-17T07:25:36.530 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:36 smithi044 ceph-mon[60356]: osdmap e124: 8 total, 8 up, 8 in 2024-09-17T07:25:36.531 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:36 smithi044 ceph-mon[60356]: pgmap v242: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:25:36.531 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:36 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/2669037623' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6801/2448217567"}]: dispatch 2024-09-17T07:25:36.531 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:36 smithi044 ceph-mon[60356]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6801/2448217567"}]: dispatch 2024-09-17T07:25:36.531 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:36 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:36.531 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:36 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:36.531 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:36 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:25:36.531 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:36 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:25:36.531 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:36 smithi044 ceph-mon[60356]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:25:36.531 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:36 smithi044 ceph-mon[60356]: pgmap v243: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-17T07:25:36.531 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:36 smithi044 ceph-mon[60356]: pgmap v244: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-09-17T07:25:36.531 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:36 smithi044 ceph-mon[60356]: Checking dashboard <-> RGW credentials 2024-09-17T07:25:36.532 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:36 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/4258601073' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/300796373"}]': finished 2024-09-17T07:25:36.532 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:36 smithi044 ceph-mon[58191]: osdmap e124: 8 total, 8 up, 8 in 2024-09-17T07:25:36.532 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:36 smithi044 ceph-mon[58191]: pgmap v242: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail 2024-09-17T07:25:36.532 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:36 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/2669037623' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6801/2448217567"}]: dispatch 2024-09-17T07:25:36.532 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:36 smithi044 ceph-mon[58191]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6801/2448217567"}]: dispatch 2024-09-17T07:25:36.532 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:36 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:36.532 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:36 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:36.532 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:36 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:25:36.532 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:36 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:25:36.532 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:36 smithi044 ceph-mon[58191]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:25:36.532 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:36 smithi044 ceph-mon[58191]: pgmap v243: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-17T07:25:36.532 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:36 smithi044 ceph-mon[58191]: pgmap v244: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-09-17T07:25:36.532 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:36 smithi044 ceph-mon[58191]: Checking dashboard <-> RGW credentials 2024-09-17T07:25:37.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:37 smithi179 ceph-mon[50085]: Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T07:25:37.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:37 smithi179 ceph-mon[50085]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6801/2448217567"}]': finished 2024-09-17T07:25:37.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:37 smithi179 ceph-mon[50085]: osdmap e125: 8 total, 8 up, 8 in 2024-09-17T07:25:37.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:37 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:37.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:37 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:37.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:37 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:37.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:37 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T07:25:37.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:37 smithi179 ceph-mon[50085]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T07:25:37.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:37 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T07:25:37.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:37 smithi179 ceph-mon[50085]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T07:25:37.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:37 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:25:37.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:37 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:37.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:37 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:37.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:37 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:37.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:37 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:37.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:37 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:37.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:37 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:37.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:37 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:37.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:37 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:37.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:37 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:37.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:37 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:37.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:37 smithi179 ceph-mon[50085]: Upgrade: Setting container_image for all iscsi 2024-09-17T07:25:37.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:37 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:37.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:37 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi.foo.smithi044.elmtfk"}]: dispatch 2024-09-17T07:25:37.367 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:37 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi.foo.smithi044.elmtfk"}]': finished 2024-09-17T07:25:37.367 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:37 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:37.367 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:37 smithi179 ceph-mon[50085]: Upgrade: Setting container_image for all nfs 2024-09-17T07:25:37.367 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:37 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:37.367 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:37 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:37.367 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:37 smithi179 ceph-mon[50085]: Upgrade: Setting container_image for all nvmeof 2024-09-17T07:25:37.367 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:37 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:37.367 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:37 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/114961699' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6800/930976792"}]: dispatch 2024-09-17T07:25:37.417 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[58191]: Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T07:25:37.417 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[58191]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6801/2448217567"}]': finished 2024-09-17T07:25:37.417 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[58191]: osdmap e125: 8 total, 8 up, 8 in 2024-09-17T07:25:37.417 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:37.417 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:37.418 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:37.418 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T07:25:37.418 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[58191]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T07:25:37.418 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T07:25:37.418 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[58191]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T07:25:37.418 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:25:37.418 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:37.418 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:37.418 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:37.418 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:37.418 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:37.418 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:37.418 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:37.418 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:37.419 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:37.419 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:37.419 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[58191]: Upgrade: Setting container_image for all iscsi 2024-09-17T07:25:37.419 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:37.419 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi.foo.smithi044.elmtfk"}]: dispatch 2024-09-17T07:25:37.419 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi.foo.smithi044.elmtfk"}]': finished 2024-09-17T07:25:37.419 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:37.419 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[58191]: Upgrade: Setting container_image for all nfs 2024-09-17T07:25:37.419 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:37.419 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:37.419 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[58191]: Upgrade: Setting container_image for all nvmeof 2024-09-17T07:25:37.419 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:37.419 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/114961699' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6800/930976792"}]: dispatch 2024-09-17T07:25:37.420 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[60356]: Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T07:25:37.420 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[60356]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6801/2448217567"}]': finished 2024-09-17T07:25:37.420 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[60356]: osdmap e125: 8 total, 8 up, 8 in 2024-09-17T07:25:37.420 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:37.420 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:37.420 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:37.420 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T07:25:37.420 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[60356]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T07:25:37.420 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T07:25:37.421 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[60356]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T07:25:37.421 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:25:37.421 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:37.421 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:37.421 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:37.421 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:37.421 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:37.421 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:37.421 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:37.421 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:37.421 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:37.421 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:37.421 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[60356]: Upgrade: Setting container_image for all iscsi 2024-09-17T07:25:37.421 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:37.422 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi.foo.smithi044.elmtfk"}]: dispatch 2024-09-17T07:25:37.422 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi.foo.smithi044.elmtfk"}]': finished 2024-09-17T07:25:37.422 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:37.422 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[60356]: Upgrade: Setting container_image for all nfs 2024-09-17T07:25:37.422 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:37.422 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:37.422 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[60356]: Upgrade: Setting container_image for all nvmeof 2024-09-17T07:25:37.422 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:37.422 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:37 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/114961699' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6800/930976792"}]: dispatch 2024-09-17T07:25:38.372 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:38 smithi044 systemd[1]: Stopping Ceph node-exporter.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:25:38.791 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:38 smithi044 podman[87408]: 2024-09-17 07:25:38.475022875 +0000 UTC m=+0.130102598 container died 8a2cde5bd9ed5da1b1583a93a47a04eba639f455650688b7108ef5d73ca66027 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a, maintainer=The Prometheus Authors ) 2024-09-17T07:25:38.792 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:38 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/114961699' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6800/930976792"}]': finished 2024-09-17T07:25:38.792 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:38 smithi044 ceph-mon[58191]: osdmap e126: 8 total, 8 up, 8 in 2024-09-17T07:25:38.792 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:38 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/3911668454' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6801/930976792"}]: dispatch 2024-09-17T07:25:38.792 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:38 smithi044 ceph-mon[58191]: Upgrade: Updating node-exporter.a (1/2) 2024-09-17T07:25:38.792 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:38 smithi044 ceph-mon[58191]: Deploying daemon node-exporter.a on smithi044 2024-09-17T07:25:38.792 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:38 smithi044 ceph-mon[58191]: pgmap v247: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-09-17T07:25:38.792 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:38 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/114961699' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6800/930976792"}]': finished 2024-09-17T07:25:38.792 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:38 smithi044 ceph-mon[60356]: osdmap e126: 8 total, 8 up, 8 in 2024-09-17T07:25:38.793 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:38 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/3911668454' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6801/930976792"}]: dispatch 2024-09-17T07:25:38.793 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:38 smithi044 ceph-mon[60356]: Upgrade: Updating node-exporter.a (1/2) 2024-09-17T07:25:38.793 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:38 smithi044 ceph-mon[60356]: Deploying daemon node-exporter.a on smithi044 2024-09-17T07:25:38.793 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:38 smithi044 ceph-mon[60356]: pgmap v247: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-09-17T07:25:38.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:38 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/114961699' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6800/930976792"}]': finished 2024-09-17T07:25:38.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:38 smithi179 ceph-mon[50085]: osdmap e126: 8 total, 8 up, 8 in 2024-09-17T07:25:38.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:38 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/3911668454' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6801/930976792"}]: dispatch 2024-09-17T07:25:38.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:38 smithi179 ceph-mon[50085]: Upgrade: Updating node-exporter.a (1/2) 2024-09-17T07:25:38.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:38 smithi179 ceph-mon[50085]: Deploying daemon node-exporter.a on smithi044 2024-09-17T07:25:38.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:38 smithi179 ceph-mon[50085]: pgmap v247: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-09-17T07:25:39.074 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:39 smithi044 podman[87408]: 2024-09-17 07:25:39.006947278 +0000 UTC m=+0.662026996 container cleanup 8a2cde5bd9ed5da1b1583a93a47a04eba639f455650688b7108ef5d73ca66027 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a, maintainer=The Prometheus Authors ) 2024-09-17T07:25:39.074 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:39 smithi044 bash[87408]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a 2024-09-17T07:25:39.703 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:39 smithi044 podman[87425]: 2024-09-17 07:25:39.325536242 +0000 UTC m=+0.853218836 container remove 8a2cde5bd9ed5da1b1583a93a47a04eba639f455650688b7108ef5d73ca66027 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.a, maintainer=The Prometheus Authors ) 2024-09-17T07:25:39.703 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:39 smithi044 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-17T07:25:39.703 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:39 smithi044 ceph-mon[58191]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:25:39.703 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:39 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/3911668454' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6801/930976792"}]': finished 2024-09-17T07:25:39.704 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:39 smithi044 ceph-mon[58191]: osdmap e127: 8 total, 8 up, 8 in 2024-09-17T07:25:39.704 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:39 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/3579926212' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/3961128045"}]: dispatch 2024-09-17T07:25:39.704 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:39 smithi044 ceph-mon[58191]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/3961128045"}]: dispatch 2024-09-17T07:25:39.704 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:39 smithi044 ceph-mon[60356]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:25:39.704 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:39 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/3911668454' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6801/930976792"}]': finished 2024-09-17T07:25:39.704 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:39 smithi044 ceph-mon[60356]: osdmap e127: 8 total, 8 up, 8 in 2024-09-17T07:25:39.704 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:39 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/3579926212' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/3961128045"}]: dispatch 2024-09-17T07:25:39.704 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:39 smithi044 ceph-mon[60356]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/3961128045"}]: dispatch 2024-09-17T07:25:39.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:39 smithi179 ceph-mon[50085]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:25:39.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:39 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/3911668454' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6801/930976792"}]': finished 2024-09-17T07:25:39.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:39 smithi179 ceph-mon[50085]: osdmap e127: 8 total, 8 up, 8 in 2024-09-17T07:25:39.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:39 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/3579926212' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/3961128045"}]: dispatch 2024-09-17T07:25:39.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:39 smithi179 ceph-mon[50085]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/3961128045"}]: dispatch 2024-09-17T07:25:40.074 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:39 smithi044 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-17T07:25:40.075 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:39 smithi044 systemd[1]: Stopped Ceph node-exporter.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:25:40.075 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:39 smithi044 systemd[1]: Starting Ceph node-exporter.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:25:40.383 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:40 smithi044 bash[87604]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-17T07:25:40.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:40 smithi044 ceph-mon[60356]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/3961128045"}]': finished 2024-09-17T07:25:40.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:40 smithi044 ceph-mon[60356]: osdmap e128: 8 total, 8 up, 8 in 2024-09-17T07:25:40.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:40 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/3916530843' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6800/3191260693"}]: dispatch 2024-09-17T07:25:40.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:40 smithi044 ceph-mon[60356]: pgmap v250: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-09-17T07:25:40.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:40 smithi044 ceph-mon[58191]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/3961128045"}]': finished 2024-09-17T07:25:40.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:40 smithi044 ceph-mon[58191]: osdmap e128: 8 total, 8 up, 8 in 2024-09-17T07:25:40.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:40 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/3916530843' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6800/3191260693"}]: dispatch 2024-09-17T07:25:40.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:40 smithi044 ceph-mon[58191]: pgmap v250: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-09-17T07:25:40.825 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:40 smithi044 bash[87604]: Getting image source signatures 2024-09-17T07:25:40.825 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:40 smithi044 bash[87604]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-17T07:25:40.825 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:40 smithi044 bash[87604]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-17T07:25:40.826 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:40 smithi044 bash[87604]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-17T07:25:40.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:40 smithi179 ceph-mon[50085]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/3961128045"}]': finished 2024-09-17T07:25:40.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:40 smithi179 ceph-mon[50085]: osdmap e128: 8 total, 8 up, 8 in 2024-09-17T07:25:40.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:40 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/3916530843' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6800/3191260693"}]: dispatch 2024-09-17T07:25:40.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:40 smithi179 ceph-mon[50085]: pgmap v250: 161 pgs: 161 active+clean; 604 KiB data, 323 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-09-17T07:25:41.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:41 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/3916530843' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6800/3191260693"}]': finished 2024-09-17T07:25:41.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:41 smithi044 ceph-mon[58191]: osdmap e129: 8 total, 8 up, 8 in 2024-09-17T07:25:41.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:41 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/321744961' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/1289983812"}]: dispatch 2024-09-17T07:25:41.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:41 smithi044 ceph-mon[58191]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/1289983812"}]: dispatch 2024-09-17T07:25:41.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:41 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/3916530843' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6800/3191260693"}]': finished 2024-09-17T07:25:41.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:41 smithi044 ceph-mon[60356]: osdmap e129: 8 total, 8 up, 8 in 2024-09-17T07:25:41.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:41 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/321744961' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/1289983812"}]: dispatch 2024-09-17T07:25:41.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:41 smithi044 ceph-mon[60356]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/1289983812"}]: dispatch 2024-09-17T07:25:41.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:41 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/3916530843' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6800/3191260693"}]': finished 2024-09-17T07:25:41.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:41 smithi179 ceph-mon[50085]: osdmap e129: 8 total, 8 up, 8 in 2024-09-17T07:25:41.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:41 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/321744961' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/1289983812"}]: dispatch 2024-09-17T07:25:41.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:41 smithi179 ceph-mon[50085]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/1289983812"}]: dispatch 2024-09-17T07:25:42.574 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 bash[87604]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-17T07:25:42.574 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 bash[87604]: Writing manifest to image destination 2024-09-17T07:25:42.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-mon[60356]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/1289983812"}]': finished 2024-09-17T07:25:42.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-mon[60356]: osdmap e130: 8 total, 8 up, 8 in 2024-09-17T07:25:42.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/3037863373' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/367718499"}]: dispatch 2024-09-17T07:25:42.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-mon[60356]: pgmap v253: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 511 B/s wr, 8 op/s 2024-09-17T07:25:42.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-mon[58191]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/1289983812"}]': finished 2024-09-17T07:25:42.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-mon[58191]: osdmap e130: 8 total, 8 up, 8 in 2024-09-17T07:25:42.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/3037863373' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/367718499"}]: dispatch 2024-09-17T07:25:42.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-mon[58191]: pgmap v253: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 511 B/s wr, 8 op/s 2024-09-17T07:25:42.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:42 smithi179 ceph-mon[50085]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/1289983812"}]': finished 2024-09-17T07:25:42.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:42 smithi179 ceph-mon[50085]: osdmap e130: 8 total, 8 up, 8 in 2024-09-17T07:25:42.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:42 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/3037863373' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/367718499"}]: dispatch 2024-09-17T07:25:42.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:42 smithi179 ceph-mon[50085]: pgmap v253: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 511 B/s wr, 8 op/s 2024-09-17T07:25:42.983 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 podman[87604]: 2024-09-17 07:25:42.622789009 +0000 UTC m=+2.591701062 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-17T07:25:42.983 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 podman[87604]: 2024-09-17 07:25:42.729952767 +0000 UTC m=+2.698864811 container create 15655da1de04d4bd939bb10394674a17bc62538fecf48a3caa145f6ce69f5763 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T07:25:42.983 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 podman[87604]: 2024-09-17 07:25:42.983063605 +0000 UTC m=+2.951975652 container init 15655da1de04d4bd939bb10394674a17bc62538fecf48a3caa145f6ce69f5763 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T07:25:43.325 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 podman[87604]: 2024-09-17 07:25:42.986165517 +0000 UTC m=+2.955077562 container start 15655da1de04d4bd939bb10394674a17bc62538fecf48a3caa145f6ce69f5763 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T07:25:43.325 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.989Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-17T07:25:43.325 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.989Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-17T07:25:43.325 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.989Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-17T07:25:43.325 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.990Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-17T07:25:43.326 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.990Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-17T07:25:43.326 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.990Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-17T07:25:43.326 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.990Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-17T07:25:43.326 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.990Z caller=node_exporter.go:117 level=info collector=arp 2024-09-17T07:25:43.326 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.990Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-17T07:25:43.326 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.990Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-17T07:25:43.326 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.990Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-17T07:25:43.326 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.990Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-17T07:25:43.326 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.990Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-17T07:25:43.326 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.990Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-17T07:25:43.326 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.990Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-17T07:25:43.326 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.990Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-17T07:25:43.327 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.990Z caller=node_exporter.go:117 level=info collector=edac 2024-09-17T07:25:43.327 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.990Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-17T07:25:43.327 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.990Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-17T07:25:43.327 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.990Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-17T07:25:43.327 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.990Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-17T07:25:43.327 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.990Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-17T07:25:43.327 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.990Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-17T07:25:43.327 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.990Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-17T07:25:43.327 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.990Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-17T07:25:43.327 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.990Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-17T07:25:43.328 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.991Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-17T07:25:43.328 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.991Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-17T07:25:43.328 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.991Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-17T07:25:43.328 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.991Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-17T07:25:43.328 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.991Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-17T07:25:43.328 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.991Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-17T07:25:43.328 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.991Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-17T07:25:43.328 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.991Z caller=node_exporter.go:117 level=info collector=os 2024-09-17T07:25:43.328 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.991Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-17T07:25:43.328 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.991Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-17T07:25:43.328 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.991Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-17T07:25:43.328 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.991Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-17T07:25:43.328 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.991Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-17T07:25:43.328 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.991Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-17T07:25:43.328 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.991Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-17T07:25:43.329 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.991Z caller=node_exporter.go:117 level=info collector=stat 2024-09-17T07:25:43.329 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.991Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-17T07:25:43.329 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.991Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-17T07:25:43.329 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.991Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-17T07:25:43.329 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.991Z caller=node_exporter.go:117 level=info collector=time 2024-09-17T07:25:43.329 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.991Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-17T07:25:43.329 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.991Z caller=node_exporter.go:117 level=info collector=uname 2024-09-17T07:25:43.329 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.991Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-17T07:25:43.329 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.991Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-17T07:25:43.329 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.991Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-17T07:25:43.329 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.991Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-17T07:25:43.329 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:42 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a[87744]: ts=2024-09-17T07:25:42.991Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-17T07:25:43.329 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:43 smithi044 bash[87604]: 15655da1de04d4bd939bb10394674a17bc62538fecf48a3caa145f6ce69f5763 2024-09-17T07:25:43.329 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:25:43 smithi044 systemd[1]: Started Ceph node-exporter.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:25:43.683 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:43 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/3037863373' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/367718499"}]': finished 2024-09-17T07:25:43.683 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:43 smithi179 ceph-mon[50085]: osdmap e131: 8 total, 8 up, 8 in 2024-09-17T07:25:43.683 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:43 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/3422400867' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/3072610708"}]: dispatch 2024-09-17T07:25:43.683 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:43 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:43.683 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:43 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:43.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:43 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/3037863373' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/367718499"}]': finished 2024-09-17T07:25:43.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:43 smithi044 ceph-mon[58191]: osdmap e131: 8 total, 8 up, 8 in 2024-09-17T07:25:43.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:43 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/3422400867' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/3072610708"}]: dispatch 2024-09-17T07:25:43.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:43 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:43.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:43 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:43.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:43 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/3037863373' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/367718499"}]': finished 2024-09-17T07:25:43.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:43 smithi044 ceph-mon[60356]: osdmap e131: 8 total, 8 up, 8 in 2024-09-17T07:25:43.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:43 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/3422400867' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/3072610708"}]: dispatch 2024-09-17T07:25:43.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:43 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:43.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:43 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:44.817 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:44 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/3422400867' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/3072610708"}]': finished 2024-09-17T07:25:44.817 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:44 smithi179 ceph-mon[50085]: osdmap e132: 8 total, 8 up, 8 in 2024-09-17T07:25:44.817 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:44 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/2784532446' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6800/3175338091"}]: dispatch 2024-09-17T07:25:44.817 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:44 smithi179 ceph-mon[50085]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6800/3175338091"}]: dispatch 2024-09-17T07:25:44.817 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:44 smithi179 ceph-mon[50085]: pgmap v256: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-17T07:25:44.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:44 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/3422400867' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/3072610708"}]': finished 2024-09-17T07:25:44.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:44 smithi044 ceph-mon[58191]: osdmap e132: 8 total, 8 up, 8 in 2024-09-17T07:25:44.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:44 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/2784532446' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6800/3175338091"}]: dispatch 2024-09-17T07:25:44.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:44 smithi044 ceph-mon[58191]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6800/3175338091"}]: dispatch 2024-09-17T07:25:44.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:44 smithi044 ceph-mon[58191]: pgmap v256: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-17T07:25:44.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:44 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/3422400867' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/3072610708"}]': finished 2024-09-17T07:25:44.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:44 smithi044 ceph-mon[60356]: osdmap e132: 8 total, 8 up, 8 in 2024-09-17T07:25:44.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:44 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/2784532446' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6800/3175338091"}]: dispatch 2024-09-17T07:25:44.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:44 smithi044 ceph-mon[60356]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6800/3175338091"}]: dispatch 2024-09-17T07:25:44.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:44 smithi044 ceph-mon[60356]: pgmap v256: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-17T07:25:45.114 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:45 smithi179 systemd[1]: Stopping Ceph node-exporter.b for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:25:45.361 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-09-17T07:25:45.548 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:45 smithi179 podman[68344]: 2024-09-17 07:25:45.259145093 +0000 UTC m=+0.213244004 container died 04c63b5cb5240428d2ca29390cca6860a9d087ba9b24dd18bb83f6957a01844e (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b, maintainer=The Prometheus Authors ) 2024-09-17T07:25:45.549 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:45 smithi179 ceph-mon[50085]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6800/3175338091"}]': finished 2024-09-17T07:25:45.549 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:45 smithi179 ceph-mon[50085]: osdmap e133: 8 total, 8 up, 8 in 2024-09-17T07:25:45.549 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:45 smithi179 ceph-mon[50085]: Upgrade: Updating node-exporter.b (2/2) 2024-09-17T07:25:45.549 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:45 smithi179 ceph-mon[50085]: Deploying daemon node-exporter.b on smithi179 2024-09-17T07:25:45.549 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:45 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/305021399' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/512688273"}]: dispatch 2024-09-17T07:25:45.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:45 smithi044 ceph-mon[58191]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6800/3175338091"}]': finished 2024-09-17T07:25:45.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:45 smithi044 ceph-mon[58191]: osdmap e133: 8 total, 8 up, 8 in 2024-09-17T07:25:45.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:45 smithi044 ceph-mon[58191]: Upgrade: Updating node-exporter.b (2/2) 2024-09-17T07:25:45.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:45 smithi044 ceph-mon[58191]: Deploying daemon node-exporter.b on smithi179 2024-09-17T07:25:45.725 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:45 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/305021399' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/512688273"}]: dispatch 2024-09-17T07:25:45.725 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:45 smithi044 ceph-mon[60356]: from='client.? ' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6800/3175338091"}]': finished 2024-09-17T07:25:45.725 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:45 smithi044 ceph-mon[60356]: osdmap e133: 8 total, 8 up, 8 in 2024-09-17T07:25:45.725 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:45 smithi044 ceph-mon[60356]: Upgrade: Updating node-exporter.b (2/2) 2024-09-17T07:25:45.726 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:45 smithi044 ceph-mon[60356]: Deploying daemon node-exporter.b on smithi179 2024-09-17T07:25:45.726 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:45 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/305021399' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/512688273"}]: dispatch 2024-09-17T07:25:45.816 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:45 smithi179 podman[68344]: 2024-09-17 07:25:45.656676438 +0000 UTC m=+0.610775328 container cleanup 04c63b5cb5240428d2ca29390cca6860a9d087ba9b24dd18bb83f6957a01844e (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b, maintainer=The Prometheus Authors ) 2024-09-17T07:25:45.816 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:45 smithi179 bash[68344]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b 2024-09-17T07:25:46.053 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T07:25:46.053 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.a smithi044 *:9093,9094 running (7m) 14s ago 12m 18.9M - 0.20.0 0881eb8f169f 0473d4b023ad 2024-09-17T07:25:46.053 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.a smithi179 *:3000 running (7m) 14s ago 11m 33.6M - 6.7.4 557c83e11646 43ab0e03dcac 2024-09-17T07:25:46.053 INFO:teuthology.orchestra.run.smithi044.stdout:iscsi.foo.smithi044.elmtfk smithi044 running (18s) 14s ago 11m 76.0M - 3.8 980a0bfcd684 cafa7e59cc83 2024-09-17T07:25:46.054 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.x smithi179 *:8443,8765 running (6m) 14s ago 16m 438M - 18.2.4-855-g19e2a2be 980a0bfcd684 66b52dc26022 2024-09-17T07:25:46.054 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.y smithi044 *:8443,8765 running (6m) 14s ago 17m 491M - 18.2.4-855-g19e2a2be 980a0bfcd684 8080dd318a1c 2024-09-17T07:25:46.054 INFO:teuthology.orchestra.run.smithi044.stdout:mon.a smithi044 running (6m) 14s ago 17m 59.1M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 c3293e8cdbba 2024-09-17T07:25:46.054 INFO:teuthology.orchestra.run.smithi044.stdout:mon.b smithi179 running (5m) 14s ago 16m 47.8M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 ade749759184 2024-09-17T07:25:46.054 INFO:teuthology.orchestra.run.smithi044.stdout:mon.c smithi044 running (5m) 14s ago 16m 48.2M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 02f85d31e0bd 2024-09-17T07:25:46.054 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.a smithi044 *:9100 starting - - - - 2024-09-17T07:25:46.054 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.b smithi179 *:9100 running (7m) 14s ago 12m 8929k - 0.18.1 e5a616e4b9cf 04c63b5cb524 2024-09-17T07:25:46.054 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (4m) 14s ago 15m 63.7M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 3ba47a4e2e95 2024-09-17T07:25:46.054 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (3m) 14s ago 15m 61.1M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 7655e43d9944 2024-09-17T07:25:46.054 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (3m) 14s ago 15m 55.8M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 43a48289eada 2024-09-17T07:25:46.054 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi044 running (2m) 14s ago 14m 62.1M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 b4323c928f39 2024-09-17T07:25:46.054 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi179 running (2m) 14s ago 14m 57.9M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 917adc8feb0b 2024-09-17T07:25:46.054 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi179 running (2m) 14s ago 14m 49.4M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 0258eb8ed0e7 2024-09-17T07:25:46.054 INFO:teuthology.orchestra.run.smithi044.stdout:osd.6 smithi179 running (94s) 14s ago 14m 50.5M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 22baeb2f62cb 2024-09-17T07:25:46.054 INFO:teuthology.orchestra.run.smithi044.stdout:osd.7 smithi179 running (62s) 14s ago 13m 56.8M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 36b85c18e11f 2024-09-17T07:25:46.054 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.a smithi179 *:9095 error 14s ago 13m - - 2024-09-17T07:25:46.055 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi044.nhyhnh smithi044 *:8000 running (46s) 14s ago 11m 74.4M - 18.2.4-855-g19e2a2be 980a0bfcd684 7cf4efc6a26b 2024-09-17T07:25:46.055 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi179.tqlzyy smithi179 *:8000 running (35s) 14s ago 11m 74.4M - 18.2.4-855-g19e2a2be 980a0bfcd684 a663c0cf2f6c 2024-09-17T07:25:46.055 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi044.vnufiz smithi044 *:80 running (41s) 14s ago 11m 76.9M - 18.2.4-855-g19e2a2be 980a0bfcd684 fe24da98468b 2024-09-17T07:25:46.055 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi179.ecjhll smithi179 *:80 running (30s) 14s ago 11m 74.3M - 18.2.4-855-g19e2a2be 980a0bfcd684 8b6adcab923f 2024-09-17T07:25:46.070 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:45 smithi179 podman[68357]: 2024-09-17 07:25:45.816345251 +0000 UTC m=+0.557534191 container remove 04c63b5cb5240428d2ca29390cca6860a9d087ba9b24dd18bb83f6957a01844e (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter.b, maintainer=The Prometheus Authors ) 2024-09-17T07:25:46.070 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:45 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-17T07:25:46.363 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:46 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-17T07:25:46.364 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:46 smithi179 systemd[1]: Stopped Ceph node-exporter.b for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:25:46.364 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:46 smithi179 systemd[1]: Starting Ceph node-exporter.b for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:25:46.468 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:25:46.468 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-09-17T07:25:46.468 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3 2024-09-17T07:25:46.469 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:25:46.469 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-09-17T07:25:46.469 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T07:25:46.469 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:25:46.469 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-09-17T07:25:46.469 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 8 2024-09-17T07:25:46.469 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:25:46.469 INFO:teuthology.orchestra.run.smithi044.stdout: "rgw": { 2024-09-17T07:25:46.469 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 4 2024-09-17T07:25:46.469 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:25:46.469 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-09-17T07:25:46.469 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 17 2024-09-17T07:25:46.469 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-09-17T07:25:46.470 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:25:46.745 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:46 smithi044 ceph-mon[60356]: from='client.55141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:25:46.745 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:46 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/305021399' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/512688273"}]': finished 2024-09-17T07:25:46.745 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:46 smithi044 ceph-mon[60356]: osdmap e134: 8 total, 8 up, 8 in 2024-09-17T07:25:46.746 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:46 smithi044 ceph-mon[60356]: from='client.55147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:25:46.746 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:46 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/376819598' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6801/3175338091"}]: dispatch 2024-09-17T07:25:46.746 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:46 smithi044 ceph-mon[60356]: pgmap v259: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:25:46.746 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:46 smithi044 ceph-mon[58191]: from='client.55141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:25:46.746 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:46 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/305021399' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/512688273"}]': finished 2024-09-17T07:25:46.746 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:46 smithi044 ceph-mon[58191]: osdmap e134: 8 total, 8 up, 8 in 2024-09-17T07:25:46.746 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:46 smithi044 ceph-mon[58191]: from='client.55147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:25:46.746 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:46 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/376819598' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6801/3175338091"}]: dispatch 2024-09-17T07:25:46.746 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:46 smithi044 ceph-mon[58191]: pgmap v259: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:25:46.807 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:46 smithi179 bash[68499]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-17T07:25:46.808 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:46 smithi179 ceph-mon[50085]: from='client.55141 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:25:46.808 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:46 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/305021399' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/512688273"}]': finished 2024-09-17T07:25:46.808 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:46 smithi179 ceph-mon[50085]: osdmap e134: 8 total, 8 up, 8 in 2024-09-17T07:25:46.808 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:46 smithi179 ceph-mon[50085]: from='client.55147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:25:46.808 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:46 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/376819598' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6801/3175338091"}]: dispatch 2024-09-17T07:25:46.808 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:46 smithi179 ceph-mon[50085]: pgmap v259: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:25:46.814 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:25:46.815 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9", 2024-09-17T07:25:46.815 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-09-17T07:25:46.815 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T07:25:46.815 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [ 2024-09-17T07:25:46.815 INFO:teuthology.orchestra.run.smithi044.stdout: "osd", 2024-09-17T07:25:46.815 INFO:teuthology.orchestra.run.smithi044.stdout: "mon", 2024-09-17T07:25:46.815 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr", 2024-09-17T07:25:46.815 INFO:teuthology.orchestra.run.smithi044.stdout: "rgw", 2024-09-17T07:25:46.815 INFO:teuthology.orchestra.run.smithi044.stdout: "iscsi" 2024-09-17T07:25:46.815 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-09-17T07:25:46.816 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "18/23 daemons upgraded", 2024-09-17T07:25:46.816 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "Currently upgrading node-exporter daemons", 2024-09-17T07:25:46.816 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-09-17T07:25:46.816 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:25:47.114 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:46 smithi179 bash[68499]: Getting image source signatures 2024-09-17T07:25:47.114 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:46 smithi179 bash[68499]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-17T07:25:47.114 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:46 smithi179 bash[68499]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-17T07:25:47.114 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:46 smithi179 bash[68499]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-17T07:25:47.217 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-09-17T07:25:47.217 INFO:teuthology.orchestra.run.smithi044.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-17T07:25:47.217 INFO:teuthology.orchestra.run.smithi044.stdout: daemon prometheus.a on smithi179 is in error state 2024-09-17T07:25:47.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:47 smithi044 ceph-mon[58191]: from='client.55159 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:25:47.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:47 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/376819598' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6801/3175338091"}]': finished 2024-09-17T07:25:47.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:47 smithi044 ceph-mon[58191]: osdmap e135: 8 total, 8 up, 8 in 2024-09-17T07:25:47.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:47 smithi044 ceph-mon[58191]: from='client.45288 172.21.15.44:0/1013389157' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:47.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:47 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/3033146198' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/3964501701"}]: dispatch 2024-09-17T07:25:47.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:47 smithi044 ceph-mon[58191]: from='client.45300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:25:47.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:47 smithi044 ceph-mon[58191]: from='client.55183 172.21.15.44:0/1229222616' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:25:47.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:47 smithi044 ceph-mon[60356]: from='client.55159 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:25:47.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:47 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/376819598' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6801/3175338091"}]': finished 2024-09-17T07:25:47.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:47 smithi044 ceph-mon[60356]: osdmap e135: 8 total, 8 up, 8 in 2024-09-17T07:25:47.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:47 smithi044 ceph-mon[60356]: from='client.45288 172.21.15.44:0/1013389157' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:47.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:47 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/3033146198' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/3964501701"}]: dispatch 2024-09-17T07:25:47.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:47 smithi044 ceph-mon[60356]: from='client.45300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:25:47.576 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:47 smithi044 ceph-mon[60356]: from='client.55183 172.21.15.44:0/1229222616' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:25:47.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:47 smithi179 ceph-mon[50085]: from='client.55159 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:25:47.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:47 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/376819598' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:6801/3175338091"}]': finished 2024-09-17T07:25:47.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:47 smithi179 ceph-mon[50085]: osdmap e135: 8 total, 8 up, 8 in 2024-09-17T07:25:47.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:47 smithi179 ceph-mon[50085]: from='client.45288 172.21.15.44:0/1013389157' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:47.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:47 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/3033146198' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/3964501701"}]: dispatch 2024-09-17T07:25:47.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:47 smithi179 ceph-mon[50085]: from='client.45300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:25:47.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:47 smithi179 ceph-mon[50085]: from='client.55183 172.21.15.44:0/1229222616' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:25:48.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:48 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/3033146198' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/3964501701"}]': finished 2024-09-17T07:25:48.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:48 smithi044 ceph-mon[58191]: osdmap e136: 8 total, 8 up, 8 in 2024-09-17T07:25:48.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:48 smithi044 ceph-mon[58191]: pgmap v262: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:25:48.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:48 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/3033146198' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/3964501701"}]': finished 2024-09-17T07:25:48.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:48 smithi044 ceph-mon[60356]: osdmap e136: 8 total, 8 up, 8 in 2024-09-17T07:25:48.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:48 smithi044 ceph-mon[60356]: pgmap v262: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:25:48.733 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:48 smithi179 bash[68499]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-17T07:25:48.733 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:48 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/3033146198' entity='client.iscsi.foo.smithi044.elmtfk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.44:0/3964501701"}]': finished 2024-09-17T07:25:48.733 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:48 smithi179 ceph-mon[50085]: osdmap e136: 8 total, 8 up, 8 in 2024-09-17T07:25:48.733 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:48 smithi179 ceph-mon[50085]: pgmap v262: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:25:49.096 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:48 smithi179 bash[68499]: Writing manifest to image destination 2024-09-17T07:25:49.348 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 podman[68499]: 2024-09-17 07:25:48.997424233 +0000 UTC m=+2.626766271 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-17T07:25:49.348 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 podman[68499]: 2024-09-17 07:25:49.112561089 +0000 UTC m=+2.741903162 container create f768630b41ebed5a4cdfc764ffc4ec3d1f655bbbc9a2e35ef77a5f968a23e0d2 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T07:25:49.348 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 podman[68499]: 2024-09-17 07:25:49.348199061 +0000 UTC m=+2.977541104 container init f768630b41ebed5a4cdfc764ffc4ec3d1f655bbbc9a2e35ef77a5f968a23e0d2 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T07:25:49.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-mon[50085]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:25:49.614 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 podman[68499]: 2024-09-17 07:25:49.351569909 +0000 UTC m=+2.980911951 container start f768630b41ebed5a4cdfc764ffc4ec3d1f655bbbc9a2e35ef77a5f968a23e0d2 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T07:25:49.614 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.356Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-17T07:25:49.615 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.356Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-17T07:25:49.615 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.357Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-17T07:25:49.615 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.357Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-17T07:25:49.615 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.357Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-17T07:25:49.615 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.357Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-17T07:25:49.615 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.357Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-17T07:25:49.615 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.357Z caller=node_exporter.go:117 level=info collector=arp 2024-09-17T07:25:49.615 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.357Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-17T07:25:49.615 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.357Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-17T07:25:49.615 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.357Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-17T07:25:49.615 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.357Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-17T07:25:49.615 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.357Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-17T07:25:49.615 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.357Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-17T07:25:49.616 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.357Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-17T07:25:49.616 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.357Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-17T07:25:49.616 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.357Z caller=node_exporter.go:117 level=info collector=edac 2024-09-17T07:25:49.616 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.357Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-17T07:25:49.616 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.357Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-17T07:25:49.616 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.358Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-17T07:25:49.616 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.358Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-17T07:25:49.616 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.358Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-17T07:25:49.616 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.358Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-17T07:25:49.616 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.358Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-17T07:25:49.616 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.358Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-17T07:25:49.616 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.358Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-17T07:25:49.616 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.358Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-17T07:25:49.617 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.358Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-17T07:25:49.617 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.358Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-17T07:25:49.617 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.358Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-17T07:25:49.617 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.358Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-17T07:25:49.617 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.358Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-17T07:25:49.617 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.358Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-17T07:25:49.617 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.358Z caller=node_exporter.go:117 level=info collector=os 2024-09-17T07:25:49.617 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.358Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-17T07:25:49.617 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.358Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-17T07:25:49.617 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.358Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-17T07:25:49.617 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.358Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-17T07:25:49.617 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.358Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-17T07:25:49.618 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.358Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-17T07:25:49.618 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.358Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-17T07:25:49.618 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.358Z caller=node_exporter.go:117 level=info collector=stat 2024-09-17T07:25:49.618 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.358Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-17T07:25:49.618 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.358Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-17T07:25:49.618 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.358Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-17T07:25:49.618 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.358Z caller=node_exporter.go:117 level=info collector=time 2024-09-17T07:25:49.618 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.358Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-17T07:25:49.618 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.358Z caller=node_exporter.go:117 level=info collector=uname 2024-09-17T07:25:49.618 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.358Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-17T07:25:49.618 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.358Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-17T07:25:49.619 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.358Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-17T07:25:49.619 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.358Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-17T07:25:49.619 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b[68577]: ts=2024-09-17T07:25:49.358Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-17T07:25:49.619 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 bash[68499]: f768630b41ebed5a4cdfc764ffc4ec3d1f655bbbc9a2e35ef77a5f968a23e0d2 2024-09-17T07:25:49.619 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:25:49 smithi179 systemd[1]: Started Ceph node-exporter.b for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:25:49.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:49 smithi044 ceph-mon[58191]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:25:49.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:49 smithi044 ceph-mon[60356]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:25:50.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:50 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:50.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:50 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:50.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:50 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:25:50.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:50 smithi044 ceph-mon[58191]: pgmap v263: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 936 B/s rd, 0 op/s 2024-09-17T07:25:50.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:50 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:50.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:50 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:50.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:50 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:25:50.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:50 smithi044 ceph-mon[60356]: pgmap v263: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 936 B/s rd, 0 op/s 2024-09-17T07:25:50.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:50 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:50.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:50 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:50.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:50 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:25:50.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:50 smithi179 ceph-mon[50085]: pgmap v263: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 936 B/s rd, 0 op/s 2024-09-17T07:25:53.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:52 smithi044 ceph-mon[58191]: pgmap v264: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-17T07:25:53.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:52 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:53.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:52 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:53.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:52 smithi044 ceph-mon[60356]: pgmap v264: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-17T07:25:53.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:52 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:53.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:52 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:53.363 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:52 smithi179 ceph-mon[50085]: pgmap v264: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-17T07:25:53.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:52 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:53.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:52 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:54.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:54 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:54.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:54 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:54.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:54 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:54.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:54 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:54.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:54 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:54.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:54 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:54.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:54 smithi044 ceph-mon[58191]: pgmap v265: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:25:54.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:54 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:54.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:54 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:54.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:54 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:54.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:54 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:54.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:54 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:54.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:54 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:54.326 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:54 smithi044 ceph-mon[60356]: pgmap v265: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:25:54.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:54 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:54.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:54 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:54.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:54 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:54.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:54 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:54.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:54 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:54.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:54 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:54.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:54 smithi179 ceph-mon[50085]: pgmap v265: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T07:25:57.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:56 smithi044 ceph-mon[58191]: pgmap v266: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:25:57.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:56 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:57.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:56 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:57.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:56 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:57.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:56 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:57.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:56 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:25:57.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:56 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:25:57.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:56 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:57.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:56 smithi044 ceph-mon[60356]: pgmap v266: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:25:57.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:56 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:57.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:56 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:57.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:56 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:57.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:56 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:57.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:56 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:25:57.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:56 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:25:57.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:56 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:57.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:56 smithi179 ceph-mon[50085]: pgmap v266: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:25:57.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:56 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:57.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:56 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:57.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:56 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:57.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:56 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:57.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:56 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:25:57.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:56 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:25:57.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:56 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:25:58.228 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:57 smithi044 ceph-mon[60356]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:25:58.228 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:57 smithi044 ceph-mon[60356]: pgmap v267: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:25:58.228 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:57 smithi044 ceph-mon[60356]: pgmap v268: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:25:58.228 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:57 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:25:58.228 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:57 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.228 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:57 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.229 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:57 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.229 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:57 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.229 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:57 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.229 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:57 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.229 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:57 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.229 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:57 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.229 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:57 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.229 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:57 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.229 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:57 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.229 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:57 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.229 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:57 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.229 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:57 smithi044 ceph-mon[58191]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:25:58.230 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:57 smithi044 ceph-mon[58191]: pgmap v267: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:25:58.230 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:57 smithi044 ceph-mon[58191]: pgmap v268: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:25:58.230 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:57 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:25:58.230 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:57 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.230 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:57 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.230 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:57 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.230 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:57 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.230 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:57 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.230 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:57 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.230 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:57 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.230 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:57 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.230 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:57 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.230 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:57 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.230 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:57 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.231 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:57 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.231 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:57 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.332 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:57 smithi179 ceph-mon[50085]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:25:58.332 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:57 smithi179 ceph-mon[50085]: pgmap v267: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:25:58.332 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:57 smithi179 ceph-mon[50085]: pgmap v268: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:25:58.332 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:57 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:25:58.333 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:57 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.333 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:57 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.333 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:57 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.333 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:57 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.333 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:57 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.333 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:57 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.333 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:57 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.333 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:57 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.333 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:57 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.333 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:57 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.334 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:57 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.334 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:57 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:25:58.334 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:57 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:00.161 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:59 smithi179 ceph-mon[50085]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:00.161 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:59 smithi179 ceph-mon[50085]: Upgrade: Updating prometheus.a 2024-09-17T07:26:00.161 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:59 smithi179 ceph-mon[50085]: Deploying daemon prometheus.a on smithi179 2024-09-17T07:26:00.161 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:25:59 smithi179 ceph-mon[50085]: pgmap v269: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:26:00.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:59 smithi044 ceph-mon[58191]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:00.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:59 smithi044 ceph-mon[58191]: Upgrade: Updating prometheus.a 2024-09-17T07:26:00.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:59 smithi044 ceph-mon[58191]: Deploying daemon prometheus.a on smithi179 2024-09-17T07:26:00.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:25:59 smithi044 ceph-mon[58191]: pgmap v269: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:26:00.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:59 smithi044 ceph-mon[60356]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:00.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:59 smithi044 ceph-mon[60356]: Upgrade: Updating prometheus.a 2024-09-17T07:26:00.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:59 smithi044 ceph-mon[60356]: Deploying daemon prometheus.a on smithi179 2024-09-17T07:26:00.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:25:59 smithi044 ceph-mon[60356]: pgmap v269: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:26:02.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:01 smithi179 ceph-mon[50085]: pgmap v270: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:26:02.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:01 smithi044 ceph-mon[58191]: pgmap v270: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:26:02.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:01 smithi044 ceph-mon[60356]: pgmap v270: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:26:02.478 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 systemd[1]: Starting Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:26:02.478 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 podman[72443]: 2024-09-17 07:26:02.23062671 +0000 UTC m=+0.016605193 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-17T07:26:02.478 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 podman[72443]: 2024-09-17 07:26:02.363475633 +0000 UTC m=+0.149454111 container create 61e60ddd30e6b6f70e3296fd4da9b046fd4c7b9d726053bbf9c25ce72cd80da7 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T07:26:02.864 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 podman[72443]: 2024-09-17 07:26:02.599726255 +0000 UTC m=+0.385704737 container init 61e60ddd30e6b6f70e3296fd4da9b046fd4c7b9d726053bbf9c25ce72cd80da7 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T07:26:02.864 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 podman[72443]: 2024-09-17 07:26:02.602516025 +0000 UTC m=+0.388494506 container start 61e60ddd30e6b6f70e3296fd4da9b046fd4c7b9d726053bbf9c25ce72cd80da7 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T07:26:02.865 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.627Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-17T07:26:02.865 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.627Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-17T07:26:02.865 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.627Z caller=main.go:570 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi179 (none))" 2024-09-17T07:26:02.865 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.627Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T07:26:02.865 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.627Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T07:26:02.865 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.629Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-17T07:26:02.865 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.629Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-17T07:26:02.865 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.629Z caller=dir_locker.go:77 level=warn component=tsdb msg="A lockfile from a previous execution already existed. It was replaced" file=/prometheus/lock 2024-09-17T07:26:02.865 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.630Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-17T07:26:02.865 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.630Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-17T07:26:02.865 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.633Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-17T07:26:02.865 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.633Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.369µs 2024-09-17T07:26:02.865 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.633Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-17T07:26:02.865 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.633Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=18 2024-09-17T07:26:02.866 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.640Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=18 2024-09-17T07:26:02.866 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.640Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=18 2024-09-17T07:26:02.866 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.640Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=18 2024-09-17T07:26:02.866 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.641Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=18 2024-09-17T07:26:02.866 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.643Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=5 maxSegment=18 2024-09-17T07:26:02.866 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.643Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=6 maxSegment=18 2024-09-17T07:26:02.866 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.643Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=7 maxSegment=18 2024-09-17T07:26:02.867 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.643Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=8 maxSegment=18 2024-09-17T07:26:02.867 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.643Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=9 maxSegment=18 2024-09-17T07:26:02.867 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.643Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=10 maxSegment=18 2024-09-17T07:26:02.867 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.644Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=11 maxSegment=18 2024-09-17T07:26:02.867 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.644Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=12 maxSegment=18 2024-09-17T07:26:02.867 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.644Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=13 maxSegment=18 2024-09-17T07:26:02.867 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.644Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=14 maxSegment=18 2024-09-17T07:26:02.867 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.644Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=15 maxSegment=18 2024-09-17T07:26:02.867 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.644Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=16 maxSegment=18 2024-09-17T07:26:02.867 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.645Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=17 maxSegment=18 2024-09-17T07:26:02.867 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.645Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=18 maxSegment=18 2024-09-17T07:26:02.867 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.645Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=55.659µs wal_replay_duration=12.219109ms wbl_replay_duration=124ns total_replay_duration=12.300813ms 2024-09-17T07:26:02.867 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.647Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-17T07:26:02.867 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.647Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-17T07:26:02.868 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.647Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T07:26:02.868 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.664Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=17.403461ms db_storage=807ns remote_storage=1.128µs web_handler=387ns query_engine=743ns scrape=425.998µs scrape_sd=88.554µs notify=16.395µs notify_sd=18.77µs rules=16.551898ms tracing=5.413µs 2024-09-17T07:26:02.868 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.664Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-17T07:26:02.868 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:26:02.664Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-17T07:26:02.868 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 bash[72443]: 61e60ddd30e6b6f70e3296fd4da9b046fd4c7b9d726053bbf9c25ce72cd80da7 2024-09-17T07:26:02.868 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:26:02 smithi179 systemd[1]: Started Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:26:04.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:03 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:04.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:03 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:04.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:03 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:26:04.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:03 smithi044 ceph-mon[60356]: pgmap v271: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:26:04.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:03 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:04.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:03 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:04.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:03 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:26:04.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:03 smithi044 ceph-mon[58191]: pgmap v271: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:26:04.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:03 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:04.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:03 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:04.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:03 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:26:04.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:03 smithi179 ceph-mon[50085]: pgmap v271: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T07:26:05.276 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:05 smithi044 ceph-mon[60356]: pgmap v272: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 567 B/s rd, 0 op/s 2024-09-17T07:26:05.276 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:05 smithi044 ceph-mon[58191]: pgmap v272: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 567 B/s rd, 0 op/s 2024-09-17T07:26:05.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:05 smithi179 ceph-mon[50085]: pgmap v272: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 567 B/s rd, 0 op/s 2024-09-17T07:26:07.471 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:07 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:07.471 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:07 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:07.471 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:07 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:07.471 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:07 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:07.472 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:07 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:07.472 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:07 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:07.472 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:07 smithi179 ceph-mon[50085]: pgmap v273: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:26:07.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:07 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:07.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:07 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:07.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:07 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:07.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:07 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:07.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:07 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:07.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:07 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:07.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:07 smithi044 ceph-mon[58191]: pgmap v273: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:26:07.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:07 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:07.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:07 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:07.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:07 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:07.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:07 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:07.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:07 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:07.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:07 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:07.826 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:07 smithi044 ceph-mon[60356]: pgmap v273: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T07:26:09.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:09 smithi179 ceph-mon[50085]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:09.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:09 smithi179 ceph-mon[50085]: pgmap v274: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:09.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:09 smithi044 ceph-mon[58191]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:09.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:09 smithi044 ceph-mon[58191]: pgmap v274: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:09.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:09 smithi044 ceph-mon[60356]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:09.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:09 smithi044 ceph-mon[60356]: pgmap v274: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:11.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:11.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:11.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:26:11.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:26:11.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[58191]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:26:11.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[58191]: pgmap v275: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 920 B/s rd, 0 op/s 2024-09-17T07:26:11.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:11.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T07:26:11.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[58191]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T07:26:11.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:26:11.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.327 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.327 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.327 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:11.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:11.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:26:11.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:26:11.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[60356]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:26:11.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[60356]: pgmap v275: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 920 B/s rd, 0 op/s 2024-09-17T07:26:11.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:11.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T07:26:11.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[60356]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T07:26:11.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:26:11.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.329 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.329 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.329 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.329 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:11 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:11 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:11.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:11 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:11.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:11 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:26:11.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:11 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:26:11.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:11 smithi179 ceph-mon[50085]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:26:11.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:11 smithi179 ceph-mon[50085]: pgmap v275: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 920 B/s rd, 0 op/s 2024-09-17T07:26:11.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:11 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:11.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:11 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T07:26:11.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:11 smithi179 ceph-mon[50085]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T07:26:11.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:11 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:26:11.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:11 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:11 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:11 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:11 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:11 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:11 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:11 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:11 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:11 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:11 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:11 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.365 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:11 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:11 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:11.366 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:11 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:12.190 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:12 smithi044 ceph-mon[60356]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-17T07:26:12.190 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:12 smithi044 ceph-mon[60356]: Cluster is now healthy 2024-09-17T07:26:12.190 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:12 smithi044 ceph-mon[60356]: Upgrade: Updating alertmanager.a 2024-09-17T07:26:12.190 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:12 smithi044 ceph-mon[60356]: Deploying daemon alertmanager.a on smithi044 2024-09-17T07:26:12.190 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:12 smithi044 ceph-mon[58191]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-17T07:26:12.191 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:12 smithi044 ceph-mon[58191]: Cluster is now healthy 2024-09-17T07:26:12.191 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:12 smithi044 ceph-mon[58191]: Upgrade: Updating alertmanager.a 2024-09-17T07:26:12.191 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:12 smithi044 ceph-mon[58191]: Deploying daemon alertmanager.a on smithi044 2024-09-17T07:26:12.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:12 smithi179 ceph-mon[50085]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-17T07:26:12.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:12 smithi179 ceph-mon[50085]: Cluster is now healthy 2024-09-17T07:26:12.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:12 smithi179 ceph-mon[50085]: Upgrade: Updating alertmanager.a 2024-09-17T07:26:12.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:12 smithi179 ceph-mon[50085]: Deploying daemon alertmanager.a on smithi044 2024-09-17T07:26:13.312 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:13 smithi044 ceph-mon[60356]: pgmap v276: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 920 B/s rd, 0 op/s 2024-09-17T07:26:13.312 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:13 smithi044 ceph-mon[58191]: pgmap v276: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 920 B/s rd, 0 op/s 2024-09-17T07:26:13.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:13 smithi179 ceph-mon[50085]: pgmap v276: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 920 B/s rd, 0 op/s 2024-09-17T07:26:15.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:15 smithi044 ceph-mon[60356]: pgmap v277: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 920 B/s rd, 0 op/s 2024-09-17T07:26:15.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:15 smithi044 ceph-mon[58191]: pgmap v277: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 920 B/s rd, 0 op/s 2024-09-17T07:26:15.325 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:26:14 smithi044 systemd[1]: Stopping Ceph alertmanager.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:26:15.325 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:26:15 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a[51867]: level=info ts=2024-09-17T07:26:15.091Z caller=main.go:536 msg="Received SIGTERM, exiting gracefully..." 2024-09-17T07:26:15.325 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:26:15 smithi044 podman[92189]: 2024-09-17 07:26:15.112476157 +0000 UTC m=+0.130063770 container died 0473d4b023adfcada42506350b000e48450c5c701f432209eb0668c0275c9080 (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a, maintainer=The Prometheus Authors ) 2024-09-17T07:26:15.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:15 smithi179 ceph-mon[50085]: pgmap v277: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 920 B/s rd, 0 op/s 2024-09-17T07:26:15.818 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:26:15 smithi044 podman[92189]: 2024-09-17 07:26:15.500194557 +0000 UTC m=+0.517782169 container cleanup 0473d4b023adfcada42506350b000e48450c5c701f432209eb0668c0275c9080 (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a, maintainer=The Prometheus Authors ) 2024-09-17T07:26:15.818 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:26:15 smithi044 bash[92189]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a 2024-09-17T07:26:16.074 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:26:15 smithi044 podman[92201]: 2024-09-17 07:26:15.818163407 +0000 UTC m=+0.704891071 container remove 0473d4b023adfcada42506350b000e48450c5c701f432209eb0668c0275c9080 (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager.a, maintainer=The Prometheus Authors ) 2024-09-17T07:26:16.074 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:26:16 smithi044 podman[92201]: 2024-09-17 07:26:16.044471954 +0000 UTC m=+0.931199618 volume remove 6de02fde8c573995b48b9c1bfe4f2d0631631890a5a56e671a890f125c3dae92 2024-09-17T07:26:16.810 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:26:16 smithi044 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-17T07:26:16.811 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:26:16 smithi044 systemd[1]: Stopped Ceph alertmanager.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:26:16.811 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:26:16 smithi044 systemd[1]: Starting Ceph alertmanager.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:26:17.075 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:26:16 smithi044 podman[92344]: 2024-09-17 07:26:16.713470042 +0000 UTC m=+0.016371095 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-09-17T07:26:17.075 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:26:16 smithi044 podman[92344]: 2024-09-17 07:26:16.83798265 +0000 UTC m=+0.140883682 volume create c6d1af4e1fe97800c61894d66ef5109babd59ef81d94a7bdeb9576e289ca2347 2024-09-17T07:26:17.075 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:26:16 smithi044 podman[92344]: 2024-09-17 07:26:16.947043007 +0000 UTC m=+0.249944084 container create 6718ecf773210f90874722265d38d2dd85c385c8b6636ae5964fc8cd4ee9f21c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T07:26:17.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:17 smithi179 ceph-mon[50085]: pgmap v278: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-17T07:26:17.370 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:17 smithi044 ceph-mon[58191]: pgmap v278: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-17T07:26:17.370 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:26:17 smithi044 podman[92344]: 2024-09-17 07:26:17.250350275 +0000 UTC m=+0.553251327 container init 6718ecf773210f90874722265d38d2dd85c385c8b6636ae5964fc8cd4ee9f21c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T07:26:17.370 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:26:17 smithi044 podman[92344]: 2024-09-17 07:26:17.254497405 +0000 UTC m=+0.557398438 container start 6718ecf773210f90874722265d38d2dd85c385c8b6636ae5964fc8cd4ee9f21c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T07:26:17.370 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:26:17 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager-a[92359]: ts=2024-09-17T07:26:17.268Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-17T07:26:17.370 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:26:17 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager-a[92359]: ts=2024-09-17T07:26:17.268Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-17T07:26:17.370 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:26:17 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager-a[92359]: ts=2024-09-17T07:26:17.270Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.44 port=9094 2024-09-17T07:26:17.370 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:26:17 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager-a[92359]: ts=2024-09-17T07:26:17.272Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-17T07:26:17.370 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:26:17 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager-a[92359]: ts=2024-09-17T07:26:17.295Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-17T07:26:17.371 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:26:17 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager-a[92359]: ts=2024-09-17T07:26:17.296Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-17T07:26:17.371 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:26:17 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager-a[92359]: ts=2024-09-17T07:26:17.299Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-17T07:26:17.371 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:26:17 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager-a[92359]: ts=2024-09-17T07:26:17.299Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-17T07:26:17.371 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:17 smithi044 ceph-mon[60356]: pgmap v278: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-17T07:26:17.597 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-09-17T07:26:17.825 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:26:17 smithi044 bash[92344]: 6718ecf773210f90874722265d38d2dd85c385c8b6636ae5964fc8cd4ee9f21c 2024-09-17T07:26:17.825 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:26:17 smithi044 systemd[1]: Started Ceph alertmanager.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:26:18.308 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T07:26:18.309 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.a smithi044 *:9093,9094 starting - - - - 2024-09-17T07:26:18.309 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.a smithi179 *:3000 running (7m) 11s ago 12m 33.7M - 6.7.4 557c83e11646 43ab0e03dcac 2024-09-17T07:26:18.309 INFO:teuthology.orchestra.run.smithi044.stdout:iscsi.foo.smithi044.elmtfk smithi044 running (50s) 11s ago 11m 52.4M - 3.8 980a0bfcd684 cafa7e59cc83 2024-09-17T07:26:18.309 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.x smithi179 *:8443,8765 running (6m) 11s ago 16m 439M - 18.2.4-855-g19e2a2be 980a0bfcd684 66b52dc26022 2024-09-17T07:26:18.309 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.y smithi044 *:8443,8765 running (7m) 11s ago 18m 493M - 18.2.4-855-g19e2a2be 980a0bfcd684 8080dd318a1c 2024-09-17T07:26:18.309 INFO:teuthology.orchestra.run.smithi044.stdout:mon.a smithi044 running (6m) 11s ago 18m 68.3M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 c3293e8cdbba 2024-09-17T07:26:18.309 INFO:teuthology.orchestra.run.smithi044.stdout:mon.b smithi179 running (5m) 11s ago 16m 50.1M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 ade749759184 2024-09-17T07:26:18.309 INFO:teuthology.orchestra.run.smithi044.stdout:mon.c smithi044 running (6m) 11s ago 17m 49.7M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 02f85d31e0bd 2024-09-17T07:26:18.309 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.a smithi044 *:9100 running (35s) 11s ago 13m 3598k - 1.5.0 0da6a335fe13 15655da1de04 2024-09-17T07:26:18.309 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.b smithi179 *:9100 running (29s) 11s ago 13m 3670k - 1.5.0 0da6a335fe13 f768630b41eb 2024-09-17T07:26:18.309 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (5m) 11s ago 16m 64.4M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 3ba47a4e2e95 2024-09-17T07:26:18.309 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (4m) 11s ago 16m 62.1M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 7655e43d9944 2024-09-17T07:26:18.309 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (3m) 11s ago 15m 57.1M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 43a48289eada 2024-09-17T07:26:18.310 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi044 running (3m) 11s ago 15m 63.1M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 b4323c928f39 2024-09-17T07:26:18.310 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi179 running (3m) 11s ago 15m 59.0M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 917adc8feb0b 2024-09-17T07:26:18.310 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi179 running (2m) 11s ago 14m 49.4M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 0258eb8ed0e7 2024-09-17T07:26:18.310 INFO:teuthology.orchestra.run.smithi044.stdout:osd.6 smithi179 running (2m) 11s ago 14m 51.5M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 22baeb2f62cb 2024-09-17T07:26:18.310 INFO:teuthology.orchestra.run.smithi044.stdout:osd.7 smithi179 running (94s) 11s ago 14m 58.1M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 36b85c18e11f 2024-09-17T07:26:18.310 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.a smithi179 *:9095 running (16s) 11s ago 13m 31.3M - 2.43.0 a07b618ecd1d 61e60ddd30e6 2024-09-17T07:26:18.310 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi044.nhyhnh smithi044 *:8000 running (79s) 11s ago 12m 74.8M - 18.2.4-855-g19e2a2be 980a0bfcd684 7cf4efc6a26b 2024-09-17T07:26:18.310 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi179.tqlzyy smithi179 *:8000 running (68s) 11s ago 12m 74.8M - 18.2.4-855-g19e2a2be 980a0bfcd684 a663c0cf2f6c 2024-09-17T07:26:18.310 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi044.vnufiz smithi044 *:80 running (73s) 11s ago 11m 77.3M - 18.2.4-855-g19e2a2be 980a0bfcd684 fe24da98468b 2024-09-17T07:26:18.310 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi179.ecjhll smithi179 *:80 running (62s) 11s ago 11m 74.6M - 18.2.4-855-g19e2a2be 980a0bfcd684 8b6adcab923f 2024-09-17T07:26:18.717 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:26:18.717 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-09-17T07:26:18.717 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3 2024-09-17T07:26:18.718 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:26:18.718 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-09-17T07:26:18.718 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T07:26:18.718 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:26:18.718 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-09-17T07:26:18.718 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 8 2024-09-17T07:26:18.718 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:26:18.718 INFO:teuthology.orchestra.run.smithi044.stdout: "rgw": { 2024-09-17T07:26:18.718 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 4 2024-09-17T07:26:18.718 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:26:18.718 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-09-17T07:26:18.718 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 17 2024-09-17T07:26:18.718 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-09-17T07:26:18.719 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:26:18.719 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:18 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:18.719 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:18 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:18.719 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:18 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:26:18.719 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:18 smithi044 ceph-mon[60356]: from='client.45312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:26:18.719 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:18 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:18.719 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:18 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:18.719 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:18 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:26:18.720 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:18 smithi044 ceph-mon[58191]: from='client.45312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:26:18.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:18 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:18.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:18 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:18.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:18 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:26:18.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:18 smithi179 ceph-mon[50085]: from='client.45312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:26:19.078 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:26:19.078 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9", 2024-09-17T07:26:19.078 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": true, 2024-09-17T07:26:19.078 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T07:26:19.078 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [ 2024-09-17T07:26:19.078 INFO:teuthology.orchestra.run.smithi044.stdout: "osd", 2024-09-17T07:26:19.078 INFO:teuthology.orchestra.run.smithi044.stdout: "mon", 2024-09-17T07:26:19.078 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr", 2024-09-17T07:26:19.078 INFO:teuthology.orchestra.run.smithi044.stdout: "rgw", 2024-09-17T07:26:19.078 INFO:teuthology.orchestra.run.smithi044.stdout: "iscsi" 2024-09-17T07:26:19.079 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-09-17T07:26:19.079 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": "18/23 daemons upgraded", 2024-09-17T07:26:19.079 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "Currently upgrading alertmanager daemons", 2024-09-17T07:26:19.079 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-09-17T07:26:19.079 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:26:19.466 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:26:19 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager-a[92359]: ts=2024-09-17T07:26:19.273Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000868569s 2024-09-17T07:26:19.482 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_OK 2024-09-17T07:26:19.803 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:19 smithi044 ceph-mon[60356]: from='client.45318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:26:19.803 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:19 smithi044 ceph-mon[60356]: pgmap v279: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 920 B/s rd, 0 op/s 2024-09-17T07:26:19.803 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:19 smithi044 ceph-mon[60356]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:19.803 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:19 smithi044 ceph-mon[60356]: from='client.55201 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:26:19.803 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:19 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/494264820' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:19.804 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:19 smithi044 ceph-mon[58191]: from='client.45318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:26:19.804 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:19 smithi044 ceph-mon[58191]: pgmap v279: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 920 B/s rd, 0 op/s 2024-09-17T07:26:19.804 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:19 smithi044 ceph-mon[58191]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:19.804 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:19 smithi044 ceph-mon[58191]: from='client.55201 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:26:19.804 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:19 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/494264820' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:19.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:19 smithi179 ceph-mon[50085]: from='client.45318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:26:19.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:19 smithi179 ceph-mon[50085]: pgmap v279: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 920 B/s rd, 0 op/s 2024-09-17T07:26:19.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:19 smithi179 ceph-mon[50085]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:19.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:19 smithi179 ceph-mon[50085]: from='client.55201 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:26:19.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:19 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/494264820' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:20.733 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:20 smithi044 ceph-mon[58191]: from='client.45336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:26:20.733 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:20 smithi044 ceph-mon[58191]: from='client.45342 172.21.15.44:0/3272772943' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:26:20.733 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:20 smithi044 ceph-mon[60356]: from='client.45336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:26:20.733 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:20 smithi044 ceph-mon[60356]: from='client.45342 172.21.15.44:0/3272772943' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:26:20.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:20 smithi179 ceph-mon[50085]: from='client.45336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:26:20.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:20 smithi179 ceph-mon[50085]: from='client.45342 172.21.15.44:0/3272772943' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:26:21.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:21 smithi044 ceph-mon[60356]: pgmap v280: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 920 B/s rd, 0 op/s 2024-09-17T07:26:21.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:21 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:21.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:21 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:21.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:21 smithi044 ceph-mon[58191]: pgmap v280: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 920 B/s rd, 0 op/s 2024-09-17T07:26:21.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:21 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:21.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:21 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:21.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:21 smithi179 ceph-mon[50085]: pgmap v280: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 920 B/s rd, 0 op/s 2024-09-17T07:26:21.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:21 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:21.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:21 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:23.023 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:22 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:23.023 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:22 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:23.024 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:22 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:23.024 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:22 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:23.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:22 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:23.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:22 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:23.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:23 smithi044 ceph-mon[58191]: pgmap v281: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:23.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:23 smithi044 ceph-mon[60356]: pgmap v281: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:24.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:23 smithi179 ceph-mon[50085]: pgmap v281: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:26.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:25 smithi179 ceph-mon[50085]: pgmap v282: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:26.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:25 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:26.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:25 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:26.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:25 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:26:26.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:25 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:26:26.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:25 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:26.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:25 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T07:26:26.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:25 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:26:26.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:25 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:25 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:25 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:25 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:25 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:25 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:25 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:25 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:25 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:25 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:25 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:25 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:25 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:25 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.116 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:25 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[58191]: pgmap v282: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:26.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:26.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:26.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:26:26.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:26:26.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:26.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T07:26:26.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:26:26.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.326 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.327 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.327 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.327 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[60356]: pgmap v282: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:26.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:26.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:26.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:26:26.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:26:26.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:26.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T07:26:26.327 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:26:26.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.328 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:26.329 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:25 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:27.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:26 smithi179 ceph-mon[50085]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:26:27.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:26 smithi179 ceph-mon[50085]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T07:26:27.276 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:26 smithi044 ceph-mon[58191]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:26:27.276 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:26 smithi044 ceph-mon[58191]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T07:26:27.277 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:26 smithi044 ceph-mon[60356]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:26:27.277 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:26 smithi044 ceph-mon[60356]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T07:26:27.575 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:26:27 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager-a[92359]: ts=2024-09-17T07:26:27.275Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003213384s 2024-09-17T07:26:28.012 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:27 smithi179 ceph-mon[50085]: pgmap v283: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:28.012 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:27 smithi179 ceph-mon[50085]: Upgrade: Updating grafana.a 2024-09-17T07:26:28.012 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:27 smithi179 ceph-mon[50085]: Deploying daemon grafana.a on smithi179 2024-09-17T07:26:28.252 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:27 smithi044 ceph-mon[60356]: pgmap v283: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:28.253 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:27 smithi044 ceph-mon[60356]: Upgrade: Updating grafana.a 2024-09-17T07:26:28.253 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:27 smithi044 ceph-mon[60356]: Deploying daemon grafana.a on smithi179 2024-09-17T07:26:28.253 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:27 smithi044 ceph-mon[58191]: pgmap v283: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:28.253 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:27 smithi044 ceph-mon[58191]: Upgrade: Updating grafana.a 2024-09-17T07:26:28.253 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:27 smithi044 ceph-mon[58191]: Deploying daemon grafana.a on smithi179 2024-09-17T07:26:29.614 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:29 smithi179 systemd[1]: Stopping Ceph grafana.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:26:29.614 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:26:29+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-09-17T07:26:29.615 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:26:29+0000 lvl=info msg="Stopped CleanUpService" logger=server reason="context canceled" 2024-09-17T07:26:29.615 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:26:29+0000 lvl=info msg="Stopped NotificationService" logger=server reason="context canceled" 2024-09-17T07:26:29.615 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:26:29+0000 lvl=info msg="Stopped provisioningServiceImpl" logger=server reason="context canceled" 2024-09-17T07:26:29.615 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:26:29+0000 lvl=info msg="Stopped UserAuthTokenService" logger=server reason="context canceled" 2024-09-17T07:26:29.615 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:26:29+0000 lvl=info msg="Stopped Stream Manager" 2024-09-17T07:26:29.615 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:26:29+0000 lvl=info msg="Stopped InternalMetricsService" logger=server reason="context canceled" 2024-09-17T07:26:29.615 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:26:29+0000 lvl=info msg="Stopped UsageStatsService" logger=server reason="context canceled" 2024-09-17T07:26:29.615 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:26:29+0000 lvl=info msg="Stopped RemoteCache" logger=server reason="context canceled" 2024-09-17T07:26:29.615 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:26:29+0000 lvl=info msg="Stopped PluginManager" logger=server reason="context canceled" 2024-09-17T07:26:29.615 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:26:29+0000 lvl=info msg="Stopped BackendPluginManager" logger=server reason="context canceled" 2024-09-17T07:26:29.615 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a[44103]: t=2024-09-17T07:26:29+0000 lvl=info msg="Stopped AlertEngine" logger=server reason="context canceled" 2024-09-17T07:26:29.615 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:29 smithi179 podman[76206]: 2024-09-17 07:26:29.340215075 +0000 UTC m=+0.135132588 container died 43ab0e03dcac946ea54f46ba796dd331f6df77f2fab1eba47fd9b42c27c42180 (image=docker.io/ceph/ceph-grafana:6.7.4, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, summary=Grafana Container configured for Ceph mgr/dashboard integration, org.label-schema.build-date=20201204, description=Ceph Grafana Container, maintainer=Paul Cuzner , io.buildah.version=1.19.8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image) 2024-09-17T07:26:29.953 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:29 smithi179 podman[76206]: 2024-09-17 07:26:29.730032164 +0000 UTC m=+0.524949670 container cleanup 43ab0e03dcac946ea54f46ba796dd331f6df77f2fab1eba47fd9b42c27c42180 (image=docker.io/ceph/ceph-grafana:6.7.4, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, io.buildah.version=1.19.8, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, org.label-schema.license=GPLv2, maintainer=Paul Cuzner , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-17T07:26:29.953 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:29 smithi179 bash[76206]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a 2024-09-17T07:26:30.239 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:29 smithi179 ceph-mon[50085]: pgmap v284: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:30.239 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:29 smithi179 ceph-mon[50085]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:30.239 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:29 smithi179 podman[76217]: 2024-09-17 07:26:29.998419691 +0000 UTC m=+0.656739768 container remove 43ab0e03dcac946ea54f46ba796dd331f6df77f2fab1eba47fd9b42c27c42180 (image=docker.io/ceph/ceph-grafana:6.7.4, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana.a, org.label-schema.name=CentOS Base Image, maintainer=Paul Cuzner , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, summary=Grafana Container configured for Ceph mgr/dashboard integration, org.label-schema.build-date=20201204, description=Ceph Grafana Container, io.buildah.version=1.19.8, org.label-schema.license=GPLv2) 2024-09-17T07:26:30.324 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:29 smithi044 ceph-mon[60356]: pgmap v284: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:30.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:29 smithi044 ceph-mon[60356]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:30.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:29 smithi044 ceph-mon[58191]: pgmap v284: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:30.325 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:29 smithi044 ceph-mon[58191]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:30.614 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:30 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@grafana.a.service: Deactivated successfully. 2024-09-17T07:26:30.614 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:30 smithi179 systemd[1]: Stopped Ceph grafana.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:26:30.614 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:30 smithi179 systemd[1]: Starting Ceph grafana.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:26:30.944 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:30 smithi179 podman[76359]: 2024-09-17 07:26:30.563576825 +0000 UTC m=+0.015349410 image pull 954c08fa618887fe9a6fe572ae541197c77e625b05f4193df7898f5ac649ada9 quay.io/ceph/ceph-grafana:9.4.7 2024-09-17T07:26:30.944 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:30 smithi179 podman[76359]: 2024-09-17 07:26:30.725297178 +0000 UTC m=+0.177069750 container create 65fcaac7b53fd4ceb79be0c207723440e3b2fa203642c5822b6e96aa3fb94ba1 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, build-date=2022-04-29T11:48:11.055897, release=239.1651231664, io.openshift.tags=base rhel8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, description=Ceph Grafana Container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.32.0, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , io.openshift.expose-services=, com.redhat.component=ubi8-container) 2024-09-17T07:26:30.945 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:30 smithi179 podman[76359]: 2024-09-17 07:26:30.944563887 +0000 UTC m=+0.396336459 container init 65fcaac7b53fd4ceb79be0c207723440e3b2fa203642c5822b6e96aa3fb94ba1 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a, name=ubi8, com.redhat.component=ubi8-container, vendor=Red Hat, Inc., release=239.1651231664, io.openshift.tags=base rhel8, distribution-scope=public, build-date=2022-04-29T11:48:11.055897, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, maintainer=Paul Cuzner , io.buildah.version=1.32.0, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container) 2024-09-17T07:26:31.195 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-mon[50085]: pgmap v285: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:31.196 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:30 smithi179 podman[76359]: 2024-09-17 07:26:30.947955497 +0000 UTC m=+0.399728070 container start 65fcaac7b53fd4ceb79be0c207723440e3b2fa203642c5822b6e96aa3fb94ba1 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a, io.openshift.expose-services=, architecture=x86_64, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, io.buildah.version=1.32.0, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-04-29T11:48:11.055897, io.k8s.display-name=Red Hat Universal Base Image 8, release=239.1651231664, version=8.5) 2024-09-17T07:26:31.196 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 bash[76359]: 65fcaac7b53fd4ceb79be0c207723440e3b2fa203642c5822b6e96aa3fb94ba1 2024-09-17T07:26:31.196 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 systemd[1]: Started Ceph grafana.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:26:31.196 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=settings t=2024-09-17T07:26:31.10152978Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2024-09-17T07:26:31.196 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=settings t=2024-09-17T07:26:31.101715013Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-09-17T07:26:31.196 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=settings t=2024-09-17T07:26:31.101734468Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-17T07:26:31.196 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=settings t=2024-09-17T07:26:31.101741723Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-17T07:26:31.196 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=settings t=2024-09-17T07:26:31.101748659Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-17T07:26:31.196 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=settings t=2024-09-17T07:26:31.101754822Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-17T07:26:31.196 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=settings t=2024-09-17T07:26:31.101760499Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-17T07:26:31.197 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=settings t=2024-09-17T07:26:31.101766275Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-17T07:26:31.197 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=settings t=2024-09-17T07:26:31.101772304Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-17T07:26:31.197 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=settings t=2024-09-17T07:26:31.101778015Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-17T07:26:31.197 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=settings t=2024-09-17T07:26:31.101783474Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-17T07:26:31.197 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=settings t=2024-09-17T07:26:31.101788892Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-17T07:26:31.197 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=settings t=2024-09-17T07:26:31.101795031Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-17T07:26:31.197 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=settings t=2024-09-17T07:26:31.101800898Z level=info msg="App mode production" 2024-09-17T07:26:31.197 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=sqlstore t=2024-09-17T07:26:31.101846862Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-17T07:26:31.197 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=sqlstore t=2024-09-17T07:26:31.101863006Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-17T07:26:31.197 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.102686983Z level=info msg="Starting DB migrations" 2024-09-17T07:26:31.197 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.102758494Z level=info msg="Executing migration" id="create migration_log table" 2024-09-17T07:26:31.197 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.10329722Z level=info msg="Executing migration" id="create user table" 2024-09-17T07:26:31.197 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.103722227Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-17T07:26:31.197 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.104192958Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-17T07:26:31.198 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.105040399Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-17T07:26:31.198 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.105978174Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-17T07:26:31.198 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.106904611Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-17T07:26:31.198 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.109886818Z level=info msg="Executing migration" id="create user table v2" 2024-09-17T07:26:31.198 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.110403147Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-17T07:26:31.198 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.113290908Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-17T07:26:31.198 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.113857795Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-17T07:26:31.198 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.116094794Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-17T07:26:31.198 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.118975834Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-17T07:26:31.198 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.121839161Z level=info msg="Executing migration" id="Update user table charset" 2024-09-17T07:26:31.199 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.124650548Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-17T07:26:31.199 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.127390664Z level=info msg="Executing migration" id="Add missing user data" 2024-09-17T07:26:31.199 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.130188901Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-17T07:26:31.199 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.133100329Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-17T07:26:31.199 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.135931279Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-17T07:26:31.199 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.139295122Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-17T07:26:31.199 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.142207556Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-17T07:26:31.199 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.145114198Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-17T07:26:31.199 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.147844055Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-17T07:26:31.200 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.150679509Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-17T07:26:31.200 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.153481679Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-17T07:26:31.200 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.153911406Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-17T07:26:31.200 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.156632958Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-17T07:26:31.200 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.159414982Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-17T07:26:31.200 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.159817818Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-17T07:26:31.200 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.162431177Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-17T07:26:31.200 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.165018554Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-17T07:26:31.200 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.167648629Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-17T07:26:31.200 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.168085829Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-17T07:26:31.200 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.17087796Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-17T07:26:31.200 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.173528797Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-17T07:26:31.200 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.176070641Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-17T07:26:31.200 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.176538287Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-17T07:26:31.201 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.179130892Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-17T07:26:31.201 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.179496794Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-17T07:26:31.201 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.181676754Z level=info msg="Executing migration" id="create star table" 2024-09-17T07:26:31.201 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.182072331Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-17T07:26:31.201 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.184341387Z level=info msg="Executing migration" id="create org table v1" 2024-09-17T07:26:31.201 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.18514282Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-17T07:26:31.201 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.186025326Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-17T07:26:31.201 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.186538551Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-17T07:26:31.201 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.187338791Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-17T07:26:31.201 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.18818101Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-17T07:26:31.201 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.189032811Z level=info msg="Executing migration" id="Update org table charset" 2024-09-17T07:26:31.201 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.189839925Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-17T07:26:31.201 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.190621789Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-17T07:26:31.202 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.191494423Z level=info msg="Executing migration" id="create dashboard table" 2024-09-17T07:26:31.202 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.192325104Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-17T07:26:31.202 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.193161436Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-17T07:26:31.202 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.193595497Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-17T07:26:31.202 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.19399886Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-17T07:26:31.202 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.194429831Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-17T07:26:31.202 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.194802127Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-17T07:26:31.202 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.195793071Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-17T07:26:31.202 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.196261298Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-17T07:26:31.202 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.196702928Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-17T07:26:31.202 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.197174389Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-17T07:26:31.202 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.197515642Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-17T07:26:31.324 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:31 smithi044 ceph-mon[58191]: pgmap v285: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:31.325 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:31 smithi044 ceph-mon[60356]: pgmap v285: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:31.447 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.197881559Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-17T07:26:31.447 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.198196942Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-17T07:26:31.447 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.19866131Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-17T07:26:31.447 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.199146782Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-17T07:26:31.447 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.199627059Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-17T07:26:31.447 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.20003112Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-17T07:26:31.447 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.200515132Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-17T07:26:31.447 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.200963182Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-17T07:26:31.447 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.201412062Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-17T07:26:31.447 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.201708522Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-17T07:26:31.447 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.20199471Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-17T07:26:31.448 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.202626206Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-17T07:26:31.448 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.20424495Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-17T07:26:31.448 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.206962212Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-17T07:26:31.448 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.20767109Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-17T07:26:31.448 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.210251056Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-17T07:26:31.448 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.210860266Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-17T07:26:31.448 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.213017979Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-17T07:26:31.448 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.215817889Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-17T07:26:31.449 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.218573671Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-17T07:26:31.449 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.221409616Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-17T07:26:31.449 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.224163197Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-17T07:26:31.449 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.224736192Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-17T07:26:31.449 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.22758564Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-17T07:26:31.449 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.230417591Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-17T07:26:31.449 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.230776646Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-17T07:26:31.449 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.233488234Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-17T07:26:31.449 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.236144644Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-17T07:26:31.449 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.238750879Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-17T07:26:31.450 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.241366318Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-17T07:26:31.450 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.244055335Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-17T07:26:31.450 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.246726019Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-17T07:26:31.450 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.249407096Z level=info msg="Executing migration" id="create data_source table" 2024-09-17T07:26:31.450 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.252070876Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-17T07:26:31.450 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.2546987Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-17T07:26:31.450 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.255157725Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-17T07:26:31.450 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.257891032Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-17T07:26:31.450 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.260471539Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-17T07:26:31.450 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.263153596Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-17T07:26:31.450 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.265898014Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-17T07:26:31.450 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.268617067Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-17T07:26:31.450 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.271230098Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-17T07:26:31.450 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.2717266Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-17T07:26:31.451 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.272506939Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-17T07:26:31.451 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.273311251Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-17T07:26:31.451 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.274110811Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-17T07:26:31.451 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.274927051Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-17T07:26:31.451 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.275753072Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-17T07:26:31.451 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.276538005Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-17T07:26:31.451 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.277388555Z level=info msg="Executing migration" id="Add uid column" 2024-09-17T07:26:31.451 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.278178941Z level=info msg="Executing migration" id="Update uid value" 2024-09-17T07:26:31.451 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.278967434Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-17T07:26:31.451 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.279755754Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-17T07:26:31.451 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.280560146Z level=info msg="Executing migration" id="create api_key table" 2024-09-17T07:26:31.451 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.281359121Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-17T07:26:31.451 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.282142197Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-17T07:26:31.451 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.282927678Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-17T07:26:31.452 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.283741483Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-17T07:26:31.452 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.284519566Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-17T07:26:31.452 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.285319999Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-17T07:26:31.452 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.286168093Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-17T07:26:31.452 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.287348296Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-17T07:26:31.452 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.287752684Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-17T07:26:31.452 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.288179471Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-17T07:26:31.452 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.288612079Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-17T07:26:31.452 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.289069399Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-17T07:26:31.452 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.289403874Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-17T07:26:31.452 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.289745976Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-17T07:26:31.452 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.290015079Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-17T07:26:31.452 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.290595015Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-17T07:26:31.452 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.291132398Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-17T07:26:31.453 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.291449997Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-17T07:26:31.453 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.291969258Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-17T07:26:31.453 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.292545607Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-17T07:26:31.453 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.292933675Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-17T07:26:31.453 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.29336139Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-17T07:26:31.453 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.293735245Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-17T07:26:31.454 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.294267209Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-17T07:26:31.454 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.294698084Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-17T07:26:31.454 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.295121642Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-17T07:26:31.454 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.295420898Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-17T07:26:31.454 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.295706507Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-17T07:26:31.454 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.296308121Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-17T07:26:31.455 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.296863218Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-17T07:26:31.455 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.297220145Z level=info msg="Executing migration" id="create quota table v1" 2024-09-17T07:26:31.455 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.297636703Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-17T07:26:31.455 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.298064671Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-17T07:26:31.455 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.298348983Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-17T07:26:31.455 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.298733535Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-17T07:26:31.455 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.299144827Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-17T07:26:31.455 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.299763618Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-17T07:26:31.455 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.300160004Z level=info msg="Executing migration" id="create session table" 2024-09-17T07:26:31.455 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.30060026Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-17T07:26:31.455 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.300872073Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-17T07:26:31.455 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.301219873Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-17T07:26:31.455 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.301616793Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-17T07:26:31.456 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.302028115Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-17T07:26:31.456 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.302326212Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-17T07:26:31.456 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.302587675Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-17T07:26:31.456 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.302863932Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-17T07:26:31.456 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.303176411Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-17T07:26:31.456 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.303591441Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-17T07:26:31.456 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.303862802Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-17T07:26:31.456 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.304561402Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-17T07:26:31.456 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.305012483Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-17T07:26:31.456 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.305602435Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-17T07:26:31.456 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.306370097Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-17T07:26:31.456 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.306799088Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-17T07:26:31.456 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.307380878Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-17T07:26:31.457 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.308021854Z level=info msg="Executing migration" id="create alert table v1" 2024-09-17T07:26:31.457 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.308631162Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-17T07:26:31.457 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.309244558Z level=info msg="Executing migration" id="add index alert state" 2024-09-17T07:26:31.457 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.310022181Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-17T07:26:31.457 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.31050812Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-17T07:26:31.457 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.310909007Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-17T07:26:31.457 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.31133641Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-17T07:26:31.457 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.313934154Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-17T07:26:31.457 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.315713748Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-17T07:26:31.457 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.316644801Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-17T07:26:31.457 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.319170167Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-17T07:26:31.457 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.319993168Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-17T07:26:31.457 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.320852754Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-17T07:26:31.458 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.321713005Z level=info msg="Executing migration" id="Add column is_default" 2024-09-17T07:26:31.458 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.322535554Z level=info msg="Executing migration" id="Add column frequency" 2024-09-17T07:26:31.458 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.323390895Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-17T07:26:31.458 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.324247159Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-17T07:26:31.458 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.325152604Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-17T07:26:31.458 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.325994383Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-17T07:26:31.458 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.326798804Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-17T07:26:31.458 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.32764361Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-17T07:26:31.458 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.32803439Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-17T07:26:31.458 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.329083856Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-17T07:26:31.458 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.329649768Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-17T07:26:31.458 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.330033832Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-17T07:26:31.458 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.330736607Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-17T07:26:31.458 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.331584904Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-17T07:26:31.459 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.332418598Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-17T07:26:31.459 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.333198733Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-17T07:26:31.459 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.333676064Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-17T07:26:31.459 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.334496998Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-17T07:26:31.459 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.335402264Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-17T07:26:31.459 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.336265394Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-17T07:26:31.459 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.337219716Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-17T07:26:31.459 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.338027418Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-17T07:26:31.459 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.33886265Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-17T07:26:31.459 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.339324503Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-17T07:26:31.459 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.340148027Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-17T07:26:31.459 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.341023461Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-17T07:26:31.459 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.341475137Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-17T07:26:31.460 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.342306539Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-17T07:26:31.460 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.343086936Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-17T07:26:31.460 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.343932596Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-17T07:26:31.460 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.344767541Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-17T07:26:31.460 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.345615998Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-17T07:26:31.460 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.346436135Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-17T07:26:31.460 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.349311853Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-17T07:26:31.460 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.352148016Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-17T07:26:31.460 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.355068809Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-17T07:26:31.460 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.357861052Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-17T07:26:31.460 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.360673171Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-17T07:26:31.460 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.36343111Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-17T07:26:31.460 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.363798231Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-17T07:26:31.461 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.366542791Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-17T07:26:31.461 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.369305421Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-17T07:26:31.461 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.372093395Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-17T07:26:31.461 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.375059634Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-17T07:26:31.461 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.377706955Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-17T07:26:31.461 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.380481829Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-17T07:26:31.461 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.380843199Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-17T07:26:31.461 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.383726628Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-17T07:26:31.461 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.386533515Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-17T07:26:31.461 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.389276986Z level=info msg="Executing migration" id="Move region to single row" 2024-09-17T07:26:31.461 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.392081497Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-17T07:26:31.461 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.394742655Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-17T07:26:31.461 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.397209106Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-17T07:26:31.462 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.39782837Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-17T07:26:31.462 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.40049173Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-17T07:26:31.462 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.40329599Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-17T07:26:31.462 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.405892793Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-17T07:26:31.462 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.4062772Z level=info msg="Executing migration" id="create test_data table" 2024-09-17T07:26:31.462 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.409023561Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-17T07:26:31.462 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.411687246Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-17T07:26:31.462 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.414288864Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-17T07:26:31.462 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.41703053Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-17T07:26:31.462 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.419517331Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-17T07:26:31.462 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.419870848Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-17T07:26:31.462 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.422712442Z level=info msg="Executing migration" id="create team table" 2024-09-17T07:26:31.462 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.425552699Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-17T07:26:31.462 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.428248338Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-17T07:26:31.463 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.430839501Z level=info msg="Executing migration" id="create team member table" 2024-09-17T07:26:31.463 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.431706985Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-17T07:26:31.463 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.432621485Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-17T07:26:31.463 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.43340536Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-17T07:26:31.463 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.43427649Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-17T07:26:31.463 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.435360182Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-17T07:26:31.463 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.436431842Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-17T07:26:31.463 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.43754431Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-17T07:26:31.463 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.438360852Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-17T07:26:31.463 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.439172333Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-17T07:26:31.463 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.440064333Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-17T07:26:31.463 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.440890097Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-17T07:26:31.463 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.441741077Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-17T07:26:31.464 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.442512437Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-17T07:26:31.464 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.442958632Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-17T07:26:31.464 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.443852414Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-17T07:26:31.464 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.444684877Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-17T07:26:31.464 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.445535698Z level=info msg="Executing migration" id="create tag table" 2024-09-17T07:26:31.464 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.446355033Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-17T07:26:31.464 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.447221498Z level=info msg="Executing migration" id="create login attempt table" 2024-09-17T07:26:31.464 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.44765603Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-17T07:26:31.464 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.448540206Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-17T07:26:31.464 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.449392934Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-17T07:26:31.464 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.451825093Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-17T07:26:31.464 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.452668531Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-17T07:26:31.464 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.453540544Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-17T07:26:31.464 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.454372077Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-17T07:26:31.697 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.455269786Z level=info msg="Executing migration" id="create user auth table" 2024-09-17T07:26:31.697 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.456122556Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-17T07:26:31.697 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.456992262Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-17T07:26:31.697 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.457419471Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-17T07:26:31.697 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.458571219Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-17T07:26:31.698 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.459685544Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-17T07:26:31.698 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.460799529Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-17T07:26:31.698 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.461971239Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-17T07:26:31.698 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.462571207Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-17T07:26:31.698 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.463675542Z level=info msg="Executing migration" id="create server_lock table" 2024-09-17T07:26:31.698 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.464224699Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-17T07:26:31.698 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.464792743Z level=info msg="Executing migration" id="create user auth token table" 2024-09-17T07:26:31.698 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.465377582Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-17T07:26:31.698 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.466198005Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-17T07:26:31.698 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.466793361Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-17T07:26:31.698 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.46738404Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-17T07:26:31.698 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.468543694Z level=info msg="Executing migration" id="create cache_data table" 2024-09-17T07:26:31.698 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.46909614Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-17T07:26:31.698 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.469694244Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-17T07:26:31.699 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.470309617Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-17T07:26:31.699 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.470901359Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-17T07:26:31.699 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.471338069Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-17T07:26:31.699 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.47172428Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-17T07:26:31.700 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.472325316Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-17T07:26:31.700 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.472903944Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-17T07:26:31.700 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.473503357Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-17T07:26:31.700 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.473874237Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-17T07:26:31.700 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.474497496Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-17T07:26:31.700 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.475015322Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-17T07:26:31.700 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.47544606Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-17T07:26:31.700 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.475859735Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-17T07:26:31.700 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.476683238Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-17T07:26:31.700 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.477167196Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-17T07:26:31.700 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.47761754Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-17T07:26:31.700 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.478020856Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-17T07:26:31.700 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.478472801Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-17T07:26:31.700 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.479115313Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-17T07:26:31.701 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.479449241Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-17T07:26:31.701 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.479871206Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-17T07:26:31.701 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.480307108Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-17T07:26:31.701 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.480722593Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-17T07:26:31.701 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.481806719Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-17T07:26:31.701 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.482834411Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-17T07:26:31.701 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.483349008Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-17T07:26:31.701 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.483747795Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-17T07:26:31.701 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.487098566Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-17T07:26:31.701 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.490376071Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-17T07:26:31.701 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.490873397Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-17T07:26:31.701 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.491316213Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-17T07:26:31.701 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.492111212Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-17T07:26:31.702 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.492534953Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-17T07:26:31.702 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.493442176Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-17T07:26:31.702 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.494234024Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-17T07:26:31.702 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.495109019Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-17T07:26:31.702 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.496079516Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-17T07:26:31.702 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.497116398Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-17T07:26:31.702 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.497967248Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-17T07:26:31.702 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.498805958Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-17T07:26:31.702 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.499269971Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-17T07:26:31.702 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.500114732Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-17T07:26:31.703 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.500978098Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-17T07:26:31.703 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.501835748Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-17T07:26:31.703 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.502273022Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-17T07:26:31.703 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.503113735Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-17T07:26:31.703 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.505858586Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-17T07:26:31.703 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.508678647Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-17T07:26:31.703 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.511578317Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-17T07:26:31.703 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.514129559Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-17T07:26:31.703 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.514570459Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-17T07:26:31.703 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.516903622Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-17T07:26:31.703 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.51784981Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-17T07:26:31.703 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.519611436Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-17T07:26:31.703 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.522216756Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-17T07:26:31.703 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.523083003Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-17T07:26:31.703 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.525746547Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-17T07:26:31.704 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.52849286Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-17T07:26:31.704 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.531292248Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-17T07:26:31.704 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.532151464Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-17T07:26:31.704 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.53496868Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-17T07:26:31.704 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.537828982Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-17T07:26:31.704 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.54033863Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-17T07:26:31.704 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.54307356Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-17T07:26:31.704 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.546016751Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-17T07:26:31.704 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.548533505Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-17T07:26:31.704 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.551237578Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-17T07:26:31.704 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.553964715Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-17T07:26:31.704 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.556663289Z level=info msg="Executing migration" id="create alert_image table" 2024-09-17T07:26:31.704 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.559294378Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-17T07:26:31.705 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.561958195Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-17T07:26:31.705 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.562266697Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-17T07:26:31.705 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.564905814Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-17T07:26:31.705 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.56537382Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-17T07:26:31.706 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.565446193Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-17T07:26:31.706 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.565688158Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-17T07:26:31.706 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.566013337Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-17T07:26:31.706 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.566437423Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-09-17T07:26:31.706 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.566551797Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-09-17T07:26:31.706 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.566658588Z level=info msg="alerts found to migrate" alerts=0 2024-09-17T07:26:31.706 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.568392823Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-17T07:26:31.706 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.569002328Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-17T07:26:31.706 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.571052006Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-17T07:26:31.706 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.573768679Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-17T07:26:31.706 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.576302363Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-17T07:26:31.706 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.579172766Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-17T07:26:31.707 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.579636302Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-17T07:26:31.707 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.580406276Z level=info msg="Executing migration" id="create data_keys table" 2024-09-17T07:26:31.707 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.580853106Z level=info msg="Executing migration" id="create secrets table" 2024-09-17T07:26:31.707 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.581317648Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-17T07:26:31.707 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.585757054Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-17T07:26:31.707 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.586668505Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-17T07:26:31.707 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.587653742Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-17T07:26:31.707 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.591958106Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-17T07:26:31.707 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.596003548Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-17T07:26:31.707 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.596962722Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-17T07:26:31.707 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.597575595Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-17T07:26:31.707 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.597883747Z level=info msg="Executing migration" id="create permission table" 2024-09-17T07:26:31.707 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.598364599Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-17T07:26:31.707 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.599098976Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-17T07:26:31.708 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.599908049Z level=info msg="Executing migration" id="create role table" 2024-09-17T07:26:31.708 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.600687772Z level=info msg="Executing migration" id="add column display_name" 2024-09-17T07:26:31.708 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.601630523Z level=info msg="Executing migration" id="add column group_name" 2024-09-17T07:26:31.708 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.602591602Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-17T07:26:31.708 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.603492082Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-17T07:26:31.708 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.604230257Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-17T07:26:31.708 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.605081486Z level=info msg="Executing migration" id="create team role table" 2024-09-17T07:26:31.708 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.605508287Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-17T07:26:31.708 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.606363507Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-17T07:26:31.708 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.607231588Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-17T07:26:31.708 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.608067442Z level=info msg="Executing migration" id="create user role table" 2024-09-17T07:26:31.708 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.608987753Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-17T07:26:31.708 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.609564051Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-17T07:26:31.708 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.610576016Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-17T07:26:31.709 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.61125546Z level=info msg="Executing migration" id="create builtin role table" 2024-09-17T07:26:31.709 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.612253168Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-17T07:26:31.709 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.612996016Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-17T07:26:31.709 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.613999391Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-17T07:26:31.709 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.615032033Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-17T07:26:31.709 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.615934538Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-17T07:26:31.709 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.616465082Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-17T07:26:31.709 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.617365927Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-17T07:26:31.709 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.618020738Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-17T07:26:31.709 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.618429801Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-17T07:26:31.709 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.618886912Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-17T07:26:31.709 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.620074565Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-17T07:26:31.709 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.621805765Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-17T07:26:31.710 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.622271675Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-17T07:26:31.710 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.623135009Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-17T07:26:31.710 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.623986383Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-17T07:26:31.710 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.624435673Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-17T07:26:31.710 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.625309886Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-17T07:26:31.710 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.625633093Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-17T07:26:31.710 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.626148222Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-17T07:26:31.710 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.626540747Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-17T07:26:31.710 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.629186214Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-17T07:26:31.710 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.63203474Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-17T07:26:31.710 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.63498561Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-17T07:26:31.710 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.63781695Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-17T07:26:31.711 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.640817585Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-17T07:26:31.711 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.643665276Z level=info msg="Executing migration" id="add correlation config column" 2024-09-17T07:26:31.711 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.64635408Z level=info msg="Executing migration" id="create entity_events table" 2024-09-17T07:26:31.711 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.649164844Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-17T07:26:31.711 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.651990213Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-17T07:26:31.711 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.652086673Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-17T07:26:31.711 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.654907467Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-17T07:26:31.711 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.655006287Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-17T07:26:31.711 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.657644534Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-17T07:26:31.711 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.660394143Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-17T07:26:31.711 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.663212299Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-17T07:26:31.711 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.666162105Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-17T07:26:31.711 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.66907347Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-17T07:26:31.712 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.671829076Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-17T07:26:31.712 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.674473504Z level=info msg="Executing migration" id="Drop public config table" 2024-09-17T07:26:31.712 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.677052498Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-17T07:26:31.712 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.679849477Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-17T07:26:31.712 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.682535868Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-17T07:26:31.712 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.685305796Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-17T07:26:31.712 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.687838287Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-17T07:26:31.712 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.690914914Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-17T07:26:31.712 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.691907261Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-17T07:26:31.712 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.694650283Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-17T07:26:31.712 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.697213776Z level=info msg="Executing migration" id="add share column" 2024-09-17T07:26:31.712 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.699793055Z level=info msg="Executing migration" id="create default alerting folders" 2024-09-17T07:26:31.712 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.700452606Z level=info msg="Executing migration" id="create file table" 2024-09-17T07:26:31.978 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.703041312Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-17T07:26:31.978 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.705580316Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-17T07:26:31.978 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.708299352Z level=info msg="Executing migration" id="create file_meta table" 2024-09-17T07:26:31.978 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.710970933Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-17T07:26:31.978 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.713403631Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-17T07:26:31.978 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.713752093Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-17T07:26:31.978 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.714609234Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-17T07:26:31.978 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.715453392Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-17T07:26:31.978 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.716359179Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-17T07:26:31.979 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.717442578Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-17T07:26:31.979 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.718327002Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-17T07:26:31.979 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.719169828Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-17T07:26:31.979 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.72001739Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-17T07:26:31.979 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.720846586Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-17T07:26:31.979 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.721665198Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-17T07:26:31.979 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.72270762Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-17T07:26:31.979 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.723755018Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-17T07:26:31.979 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.72457757Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-17T07:26:31.979 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.735196491Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-17T07:26:31.980 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.736021178Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-17T07:26:31.980 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.736519149Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-17T07:26:31.980 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.739888284Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-17T07:26:31.980 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.74070399Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-17T07:26:31.980 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.741060567Z level=info msg="Executing migration" id="create folder table" 2024-09-17T07:26:31.980 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.741505235Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-17T07:26:31.980 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.74200011Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-17T07:26:31.980 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.742853191Z level=info msg="Executing migration" id="Update folder title length" 2024-09-17T07:26:31.980 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.743717597Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-17T07:26:31.980 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=migrator t=2024-09-17T07:26:31.745427426Z level=info msg="migrations completed" performed=485 skipped=0 duration=642.688162ms 2024-09-17T07:26:31.980 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=sqlstore t=2024-09-17T07:26:31.745862317Z level=info msg="Created default organization" 2024-09-17T07:26:31.980 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=plugin.loader t=2024-09-17T07:26:31.818621753Z level=info msg="Plugin registered" pluginID=input 2024-09-17T07:26:31.980 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=plugin.loader t=2024-09-17T07:26:31.846333598Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-09-17T07:26:31.980 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=plugin.loader t=2024-09-17T07:26:31.846363494Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-09-17T07:26:31.981 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=secrets t=2024-09-17T07:26:31.846417348Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-17T07:26:31.981 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=query_data t=2024-09-17T07:26:31.849152326Z level=info msg="Query Service initialization" 2024-09-17T07:26:31.981 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=live.push_http t=2024-09-17T07:26:31.851855256Z level=info msg="Live Push Gateway initialization" 2024-09-17T07:26:32.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:32 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:32.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:32 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:32.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:32 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:26:32.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:32 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:32.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:32 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:32.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:32 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:26:32.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:32 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:32.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:32 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:32.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:32 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:26:32.719 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=infra.usagestats.collector t=2024-09-17T07:26:32.630840472Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-17T07:26:32.719 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=provisioning.datasources t=2024-09-17T07:26:32.686879371Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-17T07:26:32.719 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=provisioning.datasources t=2024-09-17T07:26:32.719390668Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-09-17T07:26:33.114 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=provisioning.alerting t=2024-09-17T07:26:32.740962222Z level=info msg="starting to provision alerting" 2024-09-17T07:26:33.114 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=provisioning.alerting t=2024-09-17T07:26:32.740989948Z level=info msg="finished to provision alerting" 2024-09-17T07:26:33.114 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=ngalert.state.manager t=2024-09-17T07:26:32.741132704Z level=info msg="Warming state cache for startup" 2024-09-17T07:26:33.114 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=grafanaStorageLogger t=2024-09-17T07:26:32.741229236Z level=info msg="storage starting" 2024-09-17T07:26:33.114 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=http.server t=2024-09-17T07:26:32.747411304Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-17T07:26:33.115 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=ngalert.state.manager t=2024-09-17T07:26:32.75146486Z level=info msg="State cache has been initialized" states=0 duration=10.331333ms 2024-09-17T07:26:33.115 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=ticker t=2024-09-17T07:26:32.751516729Z level=info msg=starting first_tick=2024-09-17T07:26:40Z 2024-09-17T07:26:33.115 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=ngalert.multiorg.alertmanager t=2024-09-17T07:26:32.75150202Z level=info msg="starting MultiOrg Alertmanager" 2024-09-17T07:26:33.115 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=sqlstore.transactions t=2024-09-17T07:26:32.752687711Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-17T07:26:33.115 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=sqlstore.transactions t=2024-09-17T07:26:32.799847745Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-17T07:26:33.115 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=sqlstore.transactions t=2024-09-17T07:26:32.811082883Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-09-17T07:26:33.115 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=sqlstore.transactions t=2024-09-17T07:26:32.822298207Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-09-17T07:26:33.115 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:26:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a[76373]: logger=sqlstore.transactions t=2024-09-17T07:26:32.860372301Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-17T07:26:33.396 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:33 smithi179 ceph-mon[50085]: pgmap v286: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:33.574 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:33 smithi044 ceph-mon[60356]: pgmap v286: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:33.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:33 smithi044 ceph-mon[58191]: pgmap v286: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:35.712 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:35 smithi179 ceph-mon[50085]: pgmap v287: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:35.712 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:35 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:35.712 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:35 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:35.712 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:35 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:35.712 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:35 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:36.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:35 smithi044 ceph-mon[58191]: pgmap v287: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:36.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:35 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:36.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:35 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:36.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:35 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:36.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:35 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:36.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:35 smithi044 ceph-mon[60356]: pgmap v287: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:36.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:35 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:36.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:35 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:36.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:35 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:36.076 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:35 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:37.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:37 smithi179 ceph-mon[50085]: pgmap v288: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:38.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:37 smithi044 ceph-mon[58191]: pgmap v288: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:38.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:37 smithi044 ceph-mon[60356]: pgmap v288: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:39.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: pgmap v289: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:39.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:39.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:39.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:39.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:26:39.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:26:39.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:26:39.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:39.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T07:26:39.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T07:26:39.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:26:39.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.576 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.577 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.577 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.577 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: pgmap v289: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:39.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:39.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:39.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:39.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:26:39.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:26:39.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:26:39.577 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:39.578 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T07:26:39.578 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T07:26:39.578 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:26:39.578 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.578 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.578 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.578 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.578 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.578 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.578 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.578 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.578 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.578 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.578 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.578 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.578 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.579 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.579 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.579 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.579 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.579 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.579 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: Upgrade: Finalizing container_image settings 2024-09-17T07:26:39.579 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:39.579 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-09-17T07:26:39.579 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-17T07:26:39.579 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T07:26:39.579 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-17T07:26:39.579 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-09-17T07:26:39.580 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-09-17T07:26:39.580 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-09-17T07:26:39.580 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-09-17T07:26:39.580 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-09-17T07:26:39.580 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-09-17T07:26:39.580 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-09-17T07:26:39.580 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-09-17T07:26:39.580 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-09-17T07:26:39.580 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-09-17T07:26:39.580 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T07:26:39.580 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-09-17T07:26:39.580 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-09-17T07:26:39.580 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-09-17T07:26:39.580 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-09-17T07:26:39.580 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-09-17T07:26:39.581 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.581 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: Upgrade: Finalizing container_image settings 2024-09-17T07:26:39.581 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:39.581 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-09-17T07:26:39.581 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-17T07:26:39.581 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T07:26:39.581 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-17T07:26:39.581 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-09-17T07:26:39.581 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-09-17T07:26:39.581 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-09-17T07:26:39.582 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-09-17T07:26:39.582 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-09-17T07:26:39.582 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-09-17T07:26:39.582 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-09-17T07:26:39.582 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-09-17T07:26:39.582 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-09-17T07:26:39.582 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-09-17T07:26:39.582 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T07:26:39.582 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-09-17T07:26:39.582 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-09-17T07:26:39.582 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-09-17T07:26:39.582 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-09-17T07:26:39.582 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-09-17T07:26:39.583 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-09-17T07:26:39.583 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-09-17T07:26:39.583 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-17T07:26:39.583 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T07:26:39.583 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T07:26:39.583 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T07:26:39.583 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T07:26:39.583 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T07:26:39.583 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T07:26:39.584 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-09-17T07:26:39.584 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-09-17T07:26:39.584 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: Upgrade: Complete! 2024-09-17T07:26:39.584 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-09-17T07:26:39.584 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-17T07:26:39.584 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:26:39.584 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:26:39.584 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:26:39.584 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:26:39.584 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:39.584 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:26:39.585 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:26:39.585 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:26:39.585 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:26:39.585 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:39.585 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-09-17T07:26:39.585 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-09-17T07:26:39.585 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-17T07:26:39.586 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T07:26:39.586 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T07:26:39.586 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T07:26:39.586 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T07:26:39.586 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T07:26:39.586 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T07:26:39.586 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-09-17T07:26:39.586 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-09-17T07:26:39.586 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: Upgrade: Complete! 2024-09-17T07:26:39.586 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-09-17T07:26:39.586 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-17T07:26:39.587 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:26:39.587 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:26:39.587 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:26:39.587 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:26:39.587 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:39.587 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:26:39.587 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:26:39.587 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:26:39.587 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:26:39.587 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:39.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: pgmap v289: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:39.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:39.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:39.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:39.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:26:39.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:26:39.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:26:39.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:39.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T07:26:39.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T07:26:39.615 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:26:39.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.616 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.617 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.617 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.617 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.617 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.617 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.617 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.617 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.617 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.617 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:39.617 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: Upgrade: Finalizing container_image settings 2024-09-17T07:26:39.617 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:39.617 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-09-17T07:26:39.618 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-17T07:26:39.618 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T07:26:39.618 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-17T07:26:39.618 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-09-17T07:26:39.618 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-09-17T07:26:39.618 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-09-17T07:26:39.618 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-09-17T07:26:39.618 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-09-17T07:26:39.618 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-09-17T07:26:39.618 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-09-17T07:26:39.618 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-09-17T07:26:39.618 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-09-17T07:26:39.618 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-09-17T07:26:39.618 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T07:26:39.619 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-09-17T07:26:39.619 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-09-17T07:26:39.619 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-09-17T07:26:39.619 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-09-17T07:26:39.619 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-09-17T07:26:39.619 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-09-17T07:26:39.619 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-09-17T07:26:39.619 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-17T07:26:39.619 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T07:26:39.619 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T07:26:39.619 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T07:26:39.619 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T07:26:39.619 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T07:26:39.619 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T07:26:39.620 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-09-17T07:26:39.620 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-09-17T07:26:39.620 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: Upgrade: Complete! 2024-09-17T07:26:39.620 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-09-17T07:26:39.620 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-17T07:26:39.620 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:26:39.620 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:26:39.620 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:26:39.620 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:26:39.620 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:39.620 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:26:39.620 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:26:39.620 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:26:39.620 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:26:39.621 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:41.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:41 smithi044 ceph-mon[58191]: pgmap v290: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:41.575 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:41 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:41.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:41 smithi044 ceph-mon[60356]: pgmap v290: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:41.575 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:41 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:41.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:41 smithi179 ceph-mon[50085]: pgmap v290: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:41.614 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:41 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:26:43.538 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:43 smithi179 ceph-mon[50085]: pgmap v291: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:43.574 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:43 smithi044 ceph-mon[58191]: pgmap v291: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:43.574 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:43 smithi044 ceph-mon[60356]: pgmap v291: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:45.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:45 smithi044 ceph-mon[58191]: pgmap v292: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:45.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:45 smithi044 ceph-mon[60356]: pgmap v292: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:45.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:45 smithi179 ceph-mon[50085]: pgmap v292: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:47.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:47 smithi044 ceph-mon[58191]: pgmap v293: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:47.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:47 smithi044 ceph-mon[60356]: pgmap v293: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:47.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:47 smithi179 ceph-mon[50085]: pgmap v293: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:49.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:49 smithi044 ceph-mon[60356]: pgmap v294: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:49.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:49 smithi044 ceph-mon[60356]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:49.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:49 smithi044 ceph-mon[58191]: pgmap v294: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:49.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:49 smithi044 ceph-mon[58191]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:49.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:49 smithi179 ceph-mon[50085]: pgmap v294: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:49.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:49 smithi179 ceph-mon[50085]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:50.575 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'ceph orch ps' 2024-09-17T07:26:50.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:50 smithi044 ceph-mon[58191]: from='client.55225 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:26:50.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:50 smithi044 ceph-mon[60356]: from='client.55225 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:26:50.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:50 smithi179 ceph-mon[50085]: from='client.55225 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:26:51.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:51 smithi044 ceph-mon[58191]: pgmap v295: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:51.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:51 smithi044 ceph-mon[60356]: pgmap v295: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:51.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:51 smithi179 ceph-mon[50085]: pgmap v295: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:52.977 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T07:26:52.978 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.a smithi044 *:9093,9094 running (36s) 31s ago 13m 17.4M - 0.25.0 c8568f914cd2 6718ecf77321 2024-09-17T07:26:52.978 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.a smithi179 *:3000 running (22s) 18s ago 13m 29.1M - 9.4.7 954c08fa6188 65fcaac7b53f 2024-09-17T07:26:52.978 INFO:teuthology.orchestra.run.smithi044.stdout:iscsi.foo.smithi044.elmtfk smithi044 running (85s) 31s ago 12m 52.6M - 3.8 980a0bfcd684 cafa7e59cc83 2024-09-17T07:26:52.978 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.x smithi179 *:8443,8765 running (7m) 18s ago 17m 439M - 18.2.4-855-g19e2a2be 980a0bfcd684 66b52dc26022 2024-09-17T07:26:52.978 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.y smithi044 *:8443,8765 running (8m) 31s ago 18m 494M - 18.2.4-855-g19e2a2be 980a0bfcd684 8080dd318a1c 2024-09-17T07:26:52.978 INFO:teuthology.orchestra.run.smithi044.stdout:mon.a smithi044 running (7m) 31s ago 18m 68.8M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 c3293e8cdbba 2024-09-17T07:26:52.978 INFO:teuthology.orchestra.run.smithi044.stdout:mon.b smithi179 running (6m) 18s ago 17m 50.9M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 ade749759184 2024-09-17T07:26:52.978 INFO:teuthology.orchestra.run.smithi044.stdout:mon.c smithi044 running (6m) 31s ago 17m 50.6M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 02f85d31e0bd 2024-09-17T07:26:52.978 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.a smithi044 *:9100 running (70s) 31s ago 13m 8024k - 1.5.0 0da6a335fe13 15655da1de04 2024-09-17T07:26:52.978 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.b smithi179 *:9100 running (63s) 18s ago 14m 7549k - 1.5.0 0da6a335fe13 f768630b41eb 2024-09-17T07:26:52.978 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (5m) 31s ago 17m 64.5M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 3ba47a4e2e95 2024-09-17T07:26:52.978 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (5m) 31s ago 16m 62.2M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 7655e43d9944 2024-09-17T07:26:52.978 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (4m) 31s ago 16m 57.2M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 43a48289eada 2024-09-17T07:26:52.978 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi044 running (4m) 31s ago 16m 63.2M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 b4323c928f39 2024-09-17T07:26:52.979 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi179 running (3m) 18s ago 15m 59.2M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 917adc8feb0b 2024-09-17T07:26:52.979 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi179 running (3m) 18s ago 15m 49.7M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 0258eb8ed0e7 2024-09-17T07:26:52.979 INFO:teuthology.orchestra.run.smithi044.stdout:osd.6 smithi179 running (2m) 18s ago 15m 51.7M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 22baeb2f62cb 2024-09-17T07:26:52.979 INFO:teuthology.orchestra.run.smithi044.stdout:osd.7 smithi179 running (2m) 18s ago 14m 58.4M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 36b85c18e11f 2024-09-17T07:26:52.979 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.a smithi179 *:9095 running (50s) 18s ago 14m 34.1M - 2.43.0 a07b618ecd1d 61e60ddd30e6 2024-09-17T07:26:52.979 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi044.nhyhnh smithi044 *:8000 running (113s) 31s ago 12m 74.9M - 18.2.4-855-g19e2a2be 980a0bfcd684 7cf4efc6a26b 2024-09-17T07:26:52.979 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi179.tqlzyy smithi179 *:8000 running (102s) 18s ago 12m 75.0M - 18.2.4-855-g19e2a2be 980a0bfcd684 a663c0cf2f6c 2024-09-17T07:26:52.979 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi044.vnufiz smithi044 *:80 running (108s) 31s ago 12m 77.4M - 18.2.4-855-g19e2a2be 980a0bfcd684 fe24da98468b 2024-09-17T07:26:52.979 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi179.ecjhll smithi179 *:80 running (97s) 18s ago 12m 74.9M - 18.2.4-855-g19e2a2be 980a0bfcd684 8b6adcab923f 2024-09-17T07:26:53.724 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:53 smithi044 ceph-mon[58191]: pgmap v296: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:53.724 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:53 smithi044 ceph-mon[60356]: pgmap v296: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:53.748 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'ceph versions' 2024-09-17T07:26:53.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:53 smithi179 ceph-mon[50085]: pgmap v296: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:54.790 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:54 smithi044 ceph-mon[60356]: from='client.45354 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:26:54.790 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:54 smithi044 ceph-mon[58191]: from='client.45354 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:26:54.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:54 smithi179 ceph-mon[50085]: from='client.45354 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:26:55.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:55 smithi044 ceph-mon[58191]: pgmap v297: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:55.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:55 smithi044 ceph-mon[60356]: pgmap v297: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:55.863 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:55 smithi179 ceph-mon[50085]: pgmap v297: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:55.915 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:26:55.916 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-09-17T07:26:55.916 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3 2024-09-17T07:26:55.916 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:26:55.916 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-09-17T07:26:55.916 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T07:26:55.916 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:26:55.916 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-09-17T07:26:55.916 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 8 2024-09-17T07:26:55.916 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:26:55.916 INFO:teuthology.orchestra.run.smithi044.stdout: "rgw": { 2024-09-17T07:26:55.916 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 4 2024-09-17T07:26:55.916 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:26:55.917 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-09-17T07:26:55.917 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 17 2024-09-17T07:26:55.917 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-09-17T07:26:55.917 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:26:56.605 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-09-17T07:26:56.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:56 smithi044 ceph-mon[58191]: from='client.45360 172.21.15.44:0/968228338' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:56.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:56 smithi044 ceph-mon[60356]: from='client.45360 172.21.15.44:0/968228338' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:56.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:56 smithi179 ceph-mon[50085]: from='client.45360 172.21.15.44:0/968228338' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:26:57.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:57 smithi044 ceph-mon[58191]: pgmap v298: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:57.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:57 smithi044 ceph-mon[60356]: pgmap v298: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:57.863 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:57 smithi179 ceph-mon[50085]: pgmap v298: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:26:58.771 INFO:teuthology.orchestra.run.smithi044.stdout:wait for servicemap items w/ changing names to refresh 2024-09-17T07:26:59.548 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'sleep 60' 2024-09-17T07:26:59.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:59 smithi044 ceph-mon[58191]: pgmap v299: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:59.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:26:59 smithi044 ceph-mon[58191]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:59.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:59 smithi044 ceph-mon[60356]: pgmap v299: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:59.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:26:59 smithi044 ceph-mon[60356]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:26:59.863 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:59 smithi179 ceph-mon[50085]: pgmap v299: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:26:59.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:26:59 smithi179 ceph-mon[50085]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:27:01.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:01 smithi044 ceph-mon[58191]: pgmap v300: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:01.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:01 smithi044 ceph-mon[60356]: pgmap v300: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:01.863 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:01 smithi179 ceph-mon[50085]: pgmap v300: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:03.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:03 smithi044 ceph-mon[58191]: pgmap v301: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:03.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:03 smithi044 ceph-mon[60356]: pgmap v301: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:03.863 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:03 smithi179 ceph-mon[50085]: pgmap v301: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:05.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:05 smithi044 ceph-mon[58191]: pgmap v302: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:05.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:05 smithi044 ceph-mon[60356]: pgmap v302: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:05.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:05 smithi179 ceph-mon[50085]: pgmap v302: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:07.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:07 smithi044 ceph-mon[58191]: pgmap v303: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:07.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:07 smithi044 ceph-mon[60356]: pgmap v303: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:07.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:07 smithi179 ceph-mon[50085]: pgmap v303: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:09.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:09 smithi044 ceph-mon[58191]: pgmap v304: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:09.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:09 smithi044 ceph-mon[58191]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:27:09.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:09 smithi044 ceph-mon[60356]: pgmap v304: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:09.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:09 smithi044 ceph-mon[60356]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:27:09.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:09 smithi179 ceph-mon[50085]: pgmap v304: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:09.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:09 smithi179 ceph-mon[50085]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:27:11.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:11 smithi044 ceph-mon[58191]: pgmap v305: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:11.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:11 smithi044 ceph-mon[60356]: pgmap v305: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:11.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:11 smithi179 ceph-mon[50085]: pgmap v305: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:13.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:13 smithi044 ceph-mon[60356]: pgmap v306: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:13.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:13 smithi044 ceph-mon[58191]: pgmap v306: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:13.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:13 smithi179 ceph-mon[50085]: pgmap v306: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:15.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:15 smithi044 ceph-mon[58191]: pgmap v307: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:15.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:15 smithi044 ceph-mon[60356]: pgmap v307: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:15.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:15 smithi179 ceph-mon[50085]: pgmap v307: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:17.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:17 smithi044 ceph-mon[58191]: pgmap v308: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:17.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:17 smithi044 ceph-mon[60356]: pgmap v308: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:17.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:17 smithi179 ceph-mon[50085]: pgmap v308: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:19.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:19 smithi044 ceph-mon[58191]: pgmap v309: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:19.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:19 smithi044 ceph-mon[58191]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:27:19.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:19 smithi044 ceph-mon[60356]: pgmap v309: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:19.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:19 smithi044 ceph-mon[60356]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:27:19.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:19 smithi179 ceph-mon[50085]: pgmap v309: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:19.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:19 smithi179 ceph-mon[50085]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:27:21.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:21 smithi044 ceph-mon[58191]: pgmap v310: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:21.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:21 smithi044 ceph-mon[60356]: pgmap v310: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:21.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:21 smithi179 ceph-mon[50085]: pgmap v310: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:23.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:23 smithi044 ceph-mon[60356]: pgmap v311: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:23.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:23 smithi044 ceph-mon[58191]: pgmap v311: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:23.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:23 smithi179 ceph-mon[50085]: pgmap v311: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:25.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:25 smithi044 ceph-mon[58191]: pgmap v312: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:25.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:25 smithi044 ceph-mon[60356]: pgmap v312: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:25.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:25 smithi179 ceph-mon[50085]: pgmap v312: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:27.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:27 smithi044 ceph-mon[60356]: pgmap v313: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:27.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:27 smithi044 ceph-mon[58191]: pgmap v313: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:27.863 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:27 smithi179 ceph-mon[50085]: pgmap v313: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:29.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:29 smithi179 ceph-mon[50085]: pgmap v314: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:29.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:29 smithi179 ceph-mon[50085]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:27:30.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:29 smithi044 ceph-mon[58191]: pgmap v314: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:30.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:29 smithi044 ceph-mon[58191]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:27:30.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:29 smithi044 ceph-mon[60356]: pgmap v314: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:30.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:29 smithi044 ceph-mon[60356]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:27:31.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:31 smithi179 ceph-mon[50085]: pgmap v315: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:32.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:31 smithi044 ceph-mon[58191]: pgmap v315: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:32.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:31 smithi044 ceph-mon[60356]: pgmap v315: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:33.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:33 smithi179 ceph-mon[50085]: pgmap v316: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:34.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:33 smithi044 ceph-mon[58191]: pgmap v316: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:34.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:33 smithi044 ceph-mon[60356]: pgmap v316: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:35.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:35 smithi179 ceph-mon[50085]: pgmap v317: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:36.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:35 smithi044 ceph-mon[60356]: pgmap v317: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:36.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:35 smithi044 ceph-mon[58191]: pgmap v317: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:37.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:37 smithi179 ceph-mon[50085]: pgmap v318: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:38.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:37 smithi044 ceph-mon[58191]: pgmap v318: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:38.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:37 smithi044 ceph-mon[60356]: pgmap v318: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:39.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:39 smithi179 ceph-mon[50085]: pgmap v319: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:39.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:39 smithi179 ceph-mon[50085]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:27:39.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:27:39.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:27:39.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:27:39.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:39 smithi179 ceph-mon[50085]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:27:40.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:39 smithi044 ceph-mon[60356]: pgmap v319: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:40.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:39 smithi044 ceph-mon[60356]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:27:40.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:27:40.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:27:40.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:27:40.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:39 smithi044 ceph-mon[60356]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:27:40.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:39 smithi044 ceph-mon[58191]: pgmap v319: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:40.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:39 smithi044 ceph-mon[58191]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:27:40.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T07:27:40.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T07:27:40.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T07:27:40.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:39 smithi044 ceph-mon[58191]: from='mgr.44106 172.21.15.44:0/136576461' entity='mgr.y' 2024-09-17T07:27:40.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:40 smithi179 ceph-mon[50085]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:27:41.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:40 smithi044 ceph-mon[58191]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:27:41.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:40 smithi044 ceph-mon[60356]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T07:27:41.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:41 smithi179 ceph-mon[50085]: pgmap v320: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:42.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:41 smithi044 ceph-mon[58191]: pgmap v320: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:42.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:41 smithi044 ceph-mon[60356]: pgmap v320: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:43.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:43 smithi179 ceph-mon[50085]: pgmap v321: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:44.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:43 smithi044 ceph-mon[58191]: pgmap v321: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:44.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:43 smithi044 ceph-mon[60356]: pgmap v321: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:45.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:45 smithi179 ceph-mon[50085]: pgmap v322: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:45.877 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:45 smithi044 ceph-mon[58191]: pgmap v322: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:45.877 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:45 smithi044 ceph-mon[60356]: pgmap v322: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:47.864 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:47 smithi179 ceph-mon[50085]: pgmap v323: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:48.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:47 smithi044 ceph-mon[58191]: pgmap v323: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:48.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:47 smithi044 ceph-mon[60356]: pgmap v323: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:50.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:49 smithi044 ceph-mon[58191]: pgmap v324: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:50.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:49 smithi044 ceph-mon[58191]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:27:50.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:49 smithi044 ceph-mon[60356]: pgmap v324: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:50.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:49 smithi044 ceph-mon[60356]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:27:50.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:49 smithi179 ceph-mon[50085]: pgmap v324: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:50.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:49 smithi179 ceph-mon[50085]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:27:52.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:51 smithi044 ceph-mon[58191]: pgmap v325: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:52.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:51 smithi044 ceph-mon[60356]: pgmap v325: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:52.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:51 smithi179 ceph-mon[50085]: pgmap v325: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:54.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:53 smithi044 ceph-mon[60356]: pgmap v326: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:54.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:53 smithi044 ceph-mon[58191]: pgmap v326: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:54.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:53 smithi179 ceph-mon[50085]: pgmap v326: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:56.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:55 smithi044 ceph-mon[58191]: pgmap v327: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:56.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:55 smithi044 ceph-mon[60356]: pgmap v327: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:56.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:55 smithi179 ceph-mon[50085]: pgmap v327: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:27:58.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:57 smithi044 ceph-mon[58191]: pgmap v328: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:58.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:57 smithi044 ceph-mon[60356]: pgmap v328: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:27:58.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:57 smithi179 ceph-mon[50085]: pgmap v328: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:28:00.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:59 smithi044 ceph-mon[58191]: pgmap v329: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:28:00.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:27:59 smithi044 ceph-mon[58191]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:28:00.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:59 smithi044 ceph-mon[60356]: pgmap v329: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:28:00.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:27:59 smithi044 ceph-mon[60356]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:28:00.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:59 smithi179 ceph-mon[50085]: pgmap v329: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:28:00.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:27:59 smithi179 ceph-mon[50085]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:28:02.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:28:01 smithi044 ceph-mon[58191]: pgmap v330: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:28:02.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:28:01 smithi044 ceph-mon[60356]: pgmap v330: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:28:02.113 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:28:01 smithi179 ceph-mon[50085]: pgmap v330: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:28:02.147 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'ceph orch ps' 2024-09-17T07:28:03.892 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:28:03 smithi044 ceph-mon[60356]: pgmap v331: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:28:03.893 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:28:03 smithi044 ceph-mon[58191]: pgmap v331: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:28:04.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:28:03 smithi179 ceph-mon[50085]: pgmap v331: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:28:04.521 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T07:28:04.521 INFO:teuthology.orchestra.run.smithi044.stdout:alertmanager.a smithi044 *:9093,9094 running (107s) 103s ago 14m 17.4M - 0.25.0 c8568f914cd2 6718ecf77321 2024-09-17T07:28:04.521 INFO:teuthology.orchestra.run.smithi044.stdout:grafana.a smithi179 *:3000 running (93s) 89s ago 14m 29.1M - 9.4.7 954c08fa6188 65fcaac7b53f 2024-09-17T07:28:04.521 INFO:teuthology.orchestra.run.smithi044.stdout:iscsi.foo.smithi044.elmtfk smithi044 running (2m) 103s ago 13m 52.6M - 3.8 980a0bfcd684 cafa7e59cc83 2024-09-17T07:28:04.521 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.x smithi179 *:8443,8765 running (8m) 89s ago 18m 439M - 18.2.4-855-g19e2a2be 980a0bfcd684 66b52dc26022 2024-09-17T07:28:04.521 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.y smithi044 *:8443,8765 running (9m) 103s ago 19m 494M - 18.2.4-855-g19e2a2be 980a0bfcd684 8080dd318a1c 2024-09-17T07:28:04.522 INFO:teuthology.orchestra.run.smithi044.stdout:mon.a smithi044 running (8m) 103s ago 20m 68.8M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 c3293e8cdbba 2024-09-17T07:28:04.522 INFO:teuthology.orchestra.run.smithi044.stdout:mon.b smithi179 running (7m) 89s ago 18m 50.9M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 ade749759184 2024-09-17T07:28:04.522 INFO:teuthology.orchestra.run.smithi044.stdout:mon.c smithi044 running (7m) 103s ago 18m 50.6M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 02f85d31e0bd 2024-09-17T07:28:04.522 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.a smithi044 *:9100 running (2m) 103s ago 14m 8024k - 1.5.0 0da6a335fe13 15655da1de04 2024-09-17T07:28:04.522 INFO:teuthology.orchestra.run.smithi044.stdout:node-exporter.b smithi179 *:9100 running (2m) 89s ago 15m 7549k - 1.5.0 0da6a335fe13 f768630b41eb 2024-09-17T07:28:04.522 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (6m) 103s ago 18m 64.5M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 3ba47a4e2e95 2024-09-17T07:28:04.522 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi044 running (6m) 103s ago 17m 62.2M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 7655e43d9944 2024-09-17T07:28:04.522 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi044 running (5m) 103s ago 17m 57.2M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 43a48289eada 2024-09-17T07:28:04.522 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi044 running (5m) 103s ago 17m 63.2M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 b4323c928f39 2024-09-17T07:28:04.522 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi179 running (4m) 89s ago 16m 59.2M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 917adc8feb0b 2024-09-17T07:28:04.522 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi179 running (4m) 89s ago 16m 49.7M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 0258eb8ed0e7 2024-09-17T07:28:04.522 INFO:teuthology.orchestra.run.smithi044.stdout:osd.6 smithi179 running (3m) 89s ago 16m 51.7M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 22baeb2f62cb 2024-09-17T07:28:04.522 INFO:teuthology.orchestra.run.smithi044.stdout:osd.7 smithi179 running (3m) 89s ago 16m 58.4M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 36b85c18e11f 2024-09-17T07:28:04.522 INFO:teuthology.orchestra.run.smithi044.stdout:prometheus.a smithi179 *:9095 running (2m) 89s ago 15m 34.1M - 2.43.0 a07b618ecd1d 61e60ddd30e6 2024-09-17T07:28:04.522 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi044.nhyhnh smithi044 *:8000 running (3m) 103s ago 13m 74.9M - 18.2.4-855-g19e2a2be 980a0bfcd684 7cf4efc6a26b 2024-09-17T07:28:04.523 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.foo.smithi179.tqlzyy smithi179 *:8000 running (2m) 89s ago 13m 75.0M - 18.2.4-855-g19e2a2be 980a0bfcd684 a663c0cf2f6c 2024-09-17T07:28:04.523 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi044.vnufiz smithi044 *:80 running (2m) 103s ago 13m 77.4M - 18.2.4-855-g19e2a2be 980a0bfcd684 fe24da98468b 2024-09-17T07:28:04.523 INFO:teuthology.orchestra.run.smithi044.stdout:rgw.smpl.smithi179.ecjhll smithi179 *:80 running (2m) 89s ago 13m 74.9M - 18.2.4-855-g19e2a2be 980a0bfcd684 8b6adcab923f 2024-09-17T07:28:05.332 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'ceph versions' 2024-09-17T07:28:05.979 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:28:05 smithi044 ceph-mon[58191]: pgmap v332: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:28:05.979 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:28:05 smithi044 ceph-mon[58191]: from='client.45366 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:28:05.980 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:28:05 smithi044 ceph-mon[60356]: pgmap v332: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:28:05.980 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:28:05 smithi044 ceph-mon[60356]: from='client.45366 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:28:06.117 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:28:05 smithi179 ceph-mon[50085]: pgmap v332: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:28:06.117 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:28:05 smithi179 ceph-mon[50085]: from='client.45366 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:28:07.494 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:28:07.494 INFO:teuthology.orchestra.run.smithi044.stdout: "mon": { 2024-09-17T07:28:07.494 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3 2024-09-17T07:28:07.494 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:28:07.494 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr": { 2024-09-17T07:28:07.494 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T07:28:07.494 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:28:07.494 INFO:teuthology.orchestra.run.smithi044.stdout: "osd": { 2024-09-17T07:28:07.494 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 8 2024-09-17T07:28:07.494 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:28:07.495 INFO:teuthology.orchestra.run.smithi044.stdout: "rgw": { 2024-09-17T07:28:07.495 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 4 2024-09-17T07:28:07.495 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-09-17T07:28:07.495 INFO:teuthology.orchestra.run.smithi044.stdout: "overall": { 2024-09-17T07:28:07.495 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 17 2024-09-17T07:28:07.495 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-09-17T07:28:07.495 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:28:07.791 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:28:07 smithi044 ceph-mon[58191]: pgmap v333: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:28:07.791 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:28:07 smithi044 ceph-mon[58191]: from='client.45372 172.21.15.44:0/3490202693' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:28:07.791 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:28:07 smithi044 ceph-mon[60356]: pgmap v333: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:28:07.791 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:28:07 smithi044 ceph-mon[60356]: from='client.45372 172.21.15.44:0/3490202693' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:28:07.916 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:28:07 smithi179 ceph-mon[50085]: pgmap v333: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:28:07.916 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:28:07 smithi179 ceph-mon[50085]: from='client.45372 172.21.15.44:0/3490202693' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:28:08.255 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'ceph orch upgrade status' 2024-09-17T07:28:09.887 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:28:09 smithi044 ceph-mon[58191]: pgmap v334: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:28:09.887 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:28:09 smithi044 ceph-mon[58191]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:28:09.887 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:28:09 smithi044 ceph-mon[60356]: pgmap v334: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:28:09.887 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:28:09 smithi044 ceph-mon[60356]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:28:10.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:28:09 smithi179 ceph-mon[50085]: pgmap v334: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:28:10.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:28:09 smithi179 ceph-mon[50085]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:28:10.400 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:28:10.400 INFO:teuthology.orchestra.run.smithi044.stdout: "target_image": null, 2024-09-17T07:28:10.400 INFO:teuthology.orchestra.run.smithi044.stdout: "in_progress": false, 2024-09-17T07:28:10.400 INFO:teuthology.orchestra.run.smithi044.stdout: "which": "", 2024-09-17T07:28:10.400 INFO:teuthology.orchestra.run.smithi044.stdout: "services_complete": [], 2024-09-17T07:28:10.401 INFO:teuthology.orchestra.run.smithi044.stdout: "progress": null, 2024-09-17T07:28:10.401 INFO:teuthology.orchestra.run.smithi044.stdout: "message": "", 2024-09-17T07:28:10.401 INFO:teuthology.orchestra.run.smithi044.stdout: "is_paused": false 2024-09-17T07:28:10.401 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:28:11.057 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'ceph health detail' 2024-09-17T07:28:12.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:28:11 smithi044 ceph-mon[60356]: pgmap v335: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:28:12.075 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:28:11 smithi044 ceph-mon[60356]: from='client.55255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:28:12.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:28:11 smithi044 ceph-mon[58191]: pgmap v335: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:28:12.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:28:11 smithi044 ceph-mon[58191]: from='client.55255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:28:12.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:28:11 smithi179 ceph-mon[50085]: pgmap v335: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:28:12.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:28:11 smithi179 ceph-mon[50085]: from='client.55255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:28:13.285 INFO:teuthology.orchestra.run.smithi044.stdout:HEALTH_OK 2024-09-17T07:28:13.948 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:28:13 smithi044 ceph-mon[58191]: pgmap v336: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:28:13.948 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:28:13 smithi044 ceph-mon[58191]: from='client.55261 172.21.15.44:0/864037694' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:28:13.948 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:28:13 smithi044 ceph-mon[60356]: pgmap v336: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:28:13.948 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:28:13 smithi044 ceph-mon[60356]: from='client.55261 172.21.15.44:0/864037694' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:28:13.973 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-09-17T07:28:14.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:28:13 smithi179 ceph-mon[50085]: pgmap v336: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:28:14.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:28:13 smithi179 ceph-mon[50085]: from='client.55261 172.21.15.44:0/864037694' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T07:28:15.952 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:28:15 smithi044 ceph-mon[58191]: pgmap v337: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:28:15.953 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:28:15 smithi044 ceph-mon[60356]: pgmap v337: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:28:16.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:28:15 smithi179 ceph-mon[50085]: pgmap v337: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:28:16.367 INFO:teuthology.orchestra.run.smithi044.stdout:true 2024-09-17T07:28:16.990 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:28:16 smithi044 ceph-mon[60356]: from='client.? 172.21.15.44:0/2962589356' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:28:16.990 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:28:16 smithi044 ceph-mon[58191]: from='client.? 172.21.15.44:0/2962589356' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:28:17.014 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-09-17T07:28:17.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:28:16 smithi179 ceph-mon[50085]: from='client.? 172.21.15.44:0/2962589356' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:28:18.030 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:28:17 smithi044 ceph-mon[58191]: pgmap v338: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:28:18.031 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:28:17 smithi044 ceph-mon[60356]: pgmap v338: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:28:18.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:28:17 smithi179 ceph-mon[50085]: pgmap v338: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:28:19.255 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)" 2024-09-17T07:28:19.848 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:28:19 smithi044 ceph-mon[58191]: pgmap v339: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:28:19.848 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:28:19 smithi044 ceph-mon[58191]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:28:19.848 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:28:19 smithi044 ceph-mon[58191]: from='client.55273 172.21.15.44:0/2142827993' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:28:19.848 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:28:19 smithi044 ceph-mon[60356]: pgmap v339: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:28:19.848 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:28:19 smithi044 ceph-mon[60356]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:28:19.848 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:28:19 smithi044 ceph-mon[60356]: from='client.55273 172.21.15.44:0/2142827993' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:28:19.873 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-09-17T07:28:20.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:28:19 smithi179 ceph-mon[50085]: pgmap v339: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:28:20.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:28:19 smithi179 ceph-mon[50085]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:28:20.115 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:28:19 smithi179 ceph-mon[50085]: from='client.55273 172.21.15.44:0/2142827993' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T07:28:21.734 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:28:21 smithi044 ceph-mon[58191]: pgmap v340: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:28:21.735 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:28:21 smithi044 ceph-mon[60356]: pgmap v340: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:28:21.974 INFO:teuthology.orchestra.run.smithi044.stdout:osd 8 2m ago - 2024-09-17T07:28:22.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:28:21 smithi179 ceph-mon[50085]: pgmap v340: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:28:22.564 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-17T07:28:22.579 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi044.front.sepia.ceph.com 2024-09-17T07:28:22.580 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- bash -c 'ceph orch upgrade ls' 2024-09-17T07:28:22.824 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:28:22 smithi044 ceph-mon[58191]: from='client.45402 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:28:22.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:28:22 smithi044 ceph-mon[60356]: from='client.45402 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:28:23.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:28:22 smithi179 ceph-mon[50085]: from='client.45402 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:28:24.032 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:28:23 smithi044 ceph-mon[58191]: pgmap v341: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:28:24.033 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:28:23 smithi044 ceph-mon[60356]: pgmap v341: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:28:24.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:28:23 smithi179 ceph-mon[50085]: pgmap v341: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:28:25.068 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-09-17T07:28:25.069 INFO:teuthology.orchestra.run.smithi044.stdout: "image": "quay.io/ceph/ceph", 2024-09-17T07:28:25.069 INFO:teuthology.orchestra.run.smithi044.stdout: "registry": "quay.io", 2024-09-17T07:28:25.069 INFO:teuthology.orchestra.run.smithi044.stdout: "bare_image": "ceph/ceph", 2024-09-17T07:28:25.069 INFO:teuthology.orchestra.run.smithi044.stdout: "versions": [ 2024-09-17T07:28:25.070 INFO:teuthology.orchestra.run.smithi044.stdout: "19.1.1", 2024-09-17T07:28:25.070 INFO:teuthology.orchestra.run.smithi044.stdout: "19.1.0", 2024-09-17T07:28:25.070 INFO:teuthology.orchestra.run.smithi044.stdout: "18.2.4", 2024-09-17T07:28:25.070 INFO:teuthology.orchestra.run.smithi044.stdout: "18.2.2", 2024-09-17T07:28:25.070 INFO:teuthology.orchestra.run.smithi044.stdout: "18.2.1", 2024-09-17T07:28:25.070 INFO:teuthology.orchestra.run.smithi044.stdout: "18.2.0" 2024-09-17T07:28:25.071 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-09-17T07:28:25.071 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-09-17T07:28:25.764 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:28:25 smithi044 ceph-mon[58191]: pgmap v342: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:28:25.764 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:28:25 smithi044 ceph-mon[58191]: from='client.45408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:28:25.764 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:28:25 smithi044 ceph-mon[60356]: pgmap v342: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:28:25.764 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:28:25 smithi044 ceph-mon[60356]: from='client.45408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:28:25.792 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0' 2024-09-17T07:28:26.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:28:25 smithi179 ceph-mon[50085]: pgmap v342: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:28:26.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:28:25 smithi179 ceph-mon[50085]: from='client.45408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:28:28.074 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:28:27 smithi044 ceph-mon[60356]: pgmap v343: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:28:28.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:28:27 smithi044 ceph-mon[58191]: pgmap v343: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:28:28.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:28:27 smithi179 ceph-mon[50085]: pgmap v343: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T07:28:28.168 INFO:teuthology.orchestra.run.smithi044.stdout: "16.2.0", 2024-09-17T07:28:28.824 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:28:28 smithi044 ceph-mon[60356]: from='client.45414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:28:28.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:28:28 smithi044 ceph-mon[58191]: from='client.45414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:28:28.882 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 817ae996-74c3-11ef-bceb-c7b262605968 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2' 2024-09-17T07:28:29.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:28:28 smithi179 ceph-mon[50085]: from='client.45414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:28:29.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:28:29 smithi044 ceph-mon[60356]: pgmap v344: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:28:29.825 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:28:29 smithi044 ceph-mon[60356]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:28:29.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:28:29 smithi044 ceph-mon[58191]: pgmap v344: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:28:29.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:28:29 smithi044 ceph-mon[58191]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:28:30.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:28:29 smithi179 ceph-mon[50085]: pgmap v344: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:28:30.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:28:29 smithi179 ceph-mon[50085]: from='client.45087 -' entity='client.iscsi.foo.smithi044.elmtfk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T07:28:31.153 INFO:teuthology.orchestra.run.smithi044.stdout: "v16.2.2", 2024-09-17T07:28:31.153 INFO:teuthology.orchestra.run.smithi044.stdout: "v16.2.2-20210505", 2024-09-17T07:28:31.803 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:28:31 smithi044 ceph-mon[58191]: pgmap v345: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:28:31.804 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:28:31 smithi044 ceph-mon[58191]: from='client.45420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:28:31.804 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:28:31 smithi044 ceph-mon[60356]: pgmap v345: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:28:31.804 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:28:31 smithi044 ceph-mon[60356]: from='client.45420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:28:31.829 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-17T07:28:31.838 INFO:tasks.cephadm:Teardown begin 2024-09-17T07:28:31.839 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-17T07:28:31.869 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-17T07:28:31.903 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-17T07:28:31.903 DEBUG:teuthology.orchestra.run.smithi044:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-17T07:28:31.926 DEBUG:teuthology.orchestra.run.smithi179:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-17T07:28:31.961 INFO:tasks.cephadm:Stopping all daemons... 2024-09-17T07:28:31.961 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-17T07:28:31.961 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-817ae996-74c3-11ef-bceb-c7b262605968@mon.a 2024-09-17T07:28:32.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:28:32 smithi044 systemd[1]: Stopping Ceph mon.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:28:32.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:28:31 smithi179 ceph-mon[50085]: pgmap v345: 161 pgs: 161 active+clean; 604 KiB data, 328 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T07:28:32.114 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:28:31 smithi179 ceph-mon[50085]: from='client.45420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T07:28:32.683 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:28:32 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon-a[58186]: 2024-09-17T07:28:32.332+0000 7fa5e47e1640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T07:28:32.683 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:28:32 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon-a[58186]: 2024-09-17T07:28:32.332+0000 7fa5e47e1640 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-09-17T07:28:32.684 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:28:32 smithi044 podman[97901]: 2024-09-17 07:28:32.353831816 +0000 UTC m=+0.195769985 container died c3293e8cdbbafc442e96bf50af8c9ceec22396bae67d61f59e1f5bae54ba9f2e (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon-a, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-17T07:28:33.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:28:32 smithi044 podman[97901]: 2024-09-17 07:28:32.857928963 +0000 UTC m=+0.699867131 container cleanup c3293e8cdbbafc442e96bf50af8c9ceec22396bae67d61f59e1f5bae54ba9f2e (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T07:28:33.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Sep 17 07:28:32 smithi044 bash[97901]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon-a 2024-09-17T07:28:33.228 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-817ae996-74c3-11ef-bceb-c7b262605968@mon.a.service' 2024-09-17T07:28:33.285 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T07:28:33.285 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-17T07:28:33.285 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-09-17T07:28:33.286 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-817ae996-74c3-11ef-bceb-c7b262605968@mon.c 2024-09-17T07:28:33.587 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:28:33 smithi044 systemd[1]: Stopping Ceph mon.c for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:28:33.897 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:28:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon-c[60352]: 2024-09-17T07:28:33.586+0000 7f5bdef19640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T07:28:33.898 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:28:33 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon-c[60352]: 2024-09-17T07:28:33.586+0000 7f5bdef19640 -1 mon.c@1(peon) e4 *** Got Signal Terminated *** 2024-09-17T07:28:33.898 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:28:33 smithi044 podman[98022]: 2024-09-17 07:28:33.618694131 +0000 UTC m=+0.142635769 container died 02f85d31e0bd0c47b0f5496a08fe10a8831fdf9a1fe2550f404c83d3c87434f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon-c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD) 2024-09-17T07:28:34.248 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:28:34 smithi044 podman[98022]: 2024-09-17 07:28:34.004532016 +0000 UTC m=+0.528473639 container cleanup 02f85d31e0bd0c47b0f5496a08fe10a8831fdf9a1fe2550f404c83d3c87434f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon-c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-17T07:28:34.248 INFO:journalctl@ceph.mon.c.smithi044.stdout:Sep 17 07:28:34 smithi044 bash[98022]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon-c 2024-09-17T07:28:34.384 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-817ae996-74c3-11ef-bceb-c7b262605968@mon.c.service' 2024-09-17T07:28:34.421 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T07:28:34.421 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-09-17T07:28:34.421 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-09-17T07:28:34.421 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-817ae996-74c3-11ef-bceb-c7b262605968@mon.b 2024-09-17T07:28:34.755 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:28:34 smithi179 systemd[1]: Stopping Ceph mon.b for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:28:35.064 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:28:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon-b[50081]: 2024-09-17T07:28:34.754+0000 7fb339050640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T07:28:35.064 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:28:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon-b[50081]: 2024-09-17T07:28:34.754+0000 7fb339050640 -1 mon.b@2(peon) e4 *** Got Signal Terminated *** 2024-09-17T07:28:35.064 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:28:34 smithi179 podman[79885]: 2024-09-17 07:28:34.785384622 +0000 UTC m=+0.168549247 container died ade749759184e8bf884522743f9f4c90c1a544720206d79d66053d8099a2a907 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon-b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b) 2024-09-17T07:28:35.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:28:35 smithi179 podman[79885]: 2024-09-17 07:28:35.246415216 +0000 UTC m=+0.629579880 container cleanup ade749759184e8bf884522743f9f4c90c1a544720206d79d66053d8099a2a907 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon-b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True) 2024-09-17T07:28:35.364 INFO:journalctl@ceph.mon.b.smithi179.stdout:Sep 17 07:28:35 smithi179 bash[79885]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-mon-b 2024-09-17T07:28:35.682 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-817ae996-74c3-11ef-bceb-c7b262605968@mon.b.service' 2024-09-17T07:28:35.731 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T07:28:35.731 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-09-17T07:28:35.731 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-17T07:28:35.731 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-817ae996-74c3-11ef-bceb-c7b262605968@mgr.y 2024-09-17T07:28:36.047 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:28:35 smithi044 systemd[1]: Stopping Ceph mgr.y for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:28:36.325 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:28:36 smithi044 podman[98142]: 2024-09-17 07:28:36.049288748 +0000 UTC m=+0.120181408 container died 8080dd318a1ca13d3238ec5b7d414656c8924e3f6cc004ec6336b373a0e78f49 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T07:28:36.819 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:28:36 smithi044 podman[98142]: 2024-09-17 07:28:36.534902355 +0000 UTC m=+0.605795014 container cleanup 8080dd318a1ca13d3238ec5b7d414656c8924e3f6cc004ec6336b373a0e78f49 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T07:28:36.820 INFO:journalctl@ceph.mgr.y.smithi044.stdout:Sep 17 07:28:36 smithi044 bash[98142]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-y 2024-09-17T07:28:36.974 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-817ae996-74c3-11ef-bceb-c7b262605968@mgr.y.service' 2024-09-17T07:28:37.010 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T07:28:37.011 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-17T07:28:37.011 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-17T07:28:37.011 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-817ae996-74c3-11ef-bceb-c7b262605968@mgr.x 2024-09-17T07:28:37.358 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:28:37 smithi179 systemd[1]: Stopping Ceph mgr.x for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:28:37.614 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:28:37 smithi179 podman[80007]: 2024-09-17 07:28:37.357487274 +0000 UTC m=+0.154320698 container died 66b52dc26022d87921e090ca1a2e81b19c1175697811cee0089a08e68f661894 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b) 2024-09-17T07:28:38.078 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:28:37 smithi179 podman[80007]: 2024-09-17 07:28:37.86836921 +0000 UTC m=+0.665202630 container cleanup 66b52dc26022d87921e090ca1a2e81b19c1175697811cee0089a08e68f661894 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b) 2024-09-17T07:28:38.078 INFO:journalctl@ceph.mgr.x.smithi179.stdout:Sep 17 07:28:37 smithi179 bash[80007]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-mgr-x 2024-09-17T07:28:38.228 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-817ae996-74c3-11ef-bceb-c7b262605968@mgr.x.service' 2024-09-17T07:28:38.263 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T07:28:38.263 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-17T07:28:38.264 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-17T07:28:38.264 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.0 2024-09-17T07:28:38.556 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:28:38 smithi044 systemd[1]: Stopping Ceph osd.0 for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:28:38.824 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:28:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0[64936]: 2024-09-17T07:28:38.555+0000 7f59cd9c0640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T07:28:38.824 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:28:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0[64936]: 2024-09-17T07:28:38.555+0000 7f59cd9c0640 -1 osd.0 136 *** Got signal Terminated *** 2024-09-17T07:28:38.825 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:28:38 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0[64936]: 2024-09-17T07:28:38.555+0000 7f59cd9c0640 -1 osd.0 136 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T07:28:44.274 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:28:44 smithi044 podman[98266]: 2024-09-17 07:28:44.001074751 +0000 UTC m=+5.537124273 container died 3ba47a4e2e95e0cfb10d02cc4b46b7ed73dfb0a322d846624c5e9ac410a83214 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True) 2024-09-17T07:28:44.574 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:28:44 smithi044 podman[98266]: 2024-09-17 07:28:44.431620745 +0000 UTC m=+5.967670246 container cleanup 3ba47a4e2e95e0cfb10d02cc4b46b7ed73dfb0a322d846624c5e9ac410a83214 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-17T07:28:44.575 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:28:44 smithi044 bash[98266]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0 2024-09-17T07:28:45.074 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:28:44 smithi044 podman[98281]: 2024-09-17 07:28:44.658393269 +0000 UTC m=+0.652176423 container remove 3ba47a4e2e95e0cfb10d02cc4b46b7ed73dfb0a322d846624c5e9ac410a83214 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-17T07:28:45.403 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:28:45 smithi044 podman[98371]: 2024-09-17 07:28:45.022081393 +0000 UTC m=+0.016508538 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:28:45.403 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:28:45 smithi044 podman[98371]: 2024-09-17 07:28:45.13428561 +0000 UTC m=+0.128712793 container create 29f34270921164f42898cc24ba9689026f65e9901d5f63b84531ca8254b07992 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-17T07:28:45.403 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:28:45 smithi044 podman[98371]: 2024-09-17 07:28:45.403516648 +0000 UTC m=+0.397943797 container init 29f34270921164f42898cc24ba9689026f65e9901d5f63b84531ca8254b07992 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-17T07:28:45.824 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:28:45 smithi044 podman[98371]: 2024-09-17 07:28:45.407562495 +0000 UTC m=+0.401989638 container start 29f34270921164f42898cc24ba9689026f65e9901d5f63b84531ca8254b07992 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T07:28:45.825 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:28:45 smithi044 podman[98371]: 2024-09-17 07:28:45.495293009 +0000 UTC m=+0.489720149 container attach 29f34270921164f42898cc24ba9689026f65e9901d5f63b84531ca8254b07992 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T07:28:45.825 INFO:journalctl@ceph.osd.0.smithi044.stdout:Sep 17 07:28:45 smithi044 podman[98371]: 2024-09-17 07:28:45.595723331 +0000 UTC m=+0.590150471 container died 29f34270921164f42898cc24ba9689026f65e9901d5f63b84531ca8254b07992 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-17T07:28:46.477 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.0.service' 2024-09-17T07:28:46.518 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T07:28:46.518 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-17T07:28:46.518 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-17T07:28:46.518 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.1 2024-09-17T07:28:46.854 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:28:46 smithi044 systemd[1]: Stopping Ceph osd.1 for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:28:47.324 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:28:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1[67904]: 2024-09-17T07:28:46.853+0000 7f0ae0fac640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T07:28:47.324 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:28:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1[67904]: 2024-09-17T07:28:46.853+0000 7f0ae0fac640 -1 osd.1 136 *** Got signal Terminated *** 2024-09-17T07:28:47.325 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:28:46 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1[67904]: 2024-09-17T07:28:46.853+0000 7f0ae0fac640 -1 osd.1 136 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T07:28:52.654 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:28:52 smithi044 podman[98486]: 2024-09-17 07:28:52.367313957 +0000 UTC m=+5.620023541 container died 7655e43d99443ab88f001c32cd5778554910f0bc7e2219421e4ac99ef5a5de5a (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1, ceph=True, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-17T07:28:52.998 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:28:52 smithi044 podman[98486]: 2024-09-17 07:28:52.753211795 +0000 UTC m=+6.005921388 container cleanup 7655e43d99443ab88f001c32cd5778554910f0bc7e2219421e4ac99ef5a5de5a (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-17T07:28:52.999 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:28:52 smithi044 bash[98486]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1 2024-09-17T07:28:53.324 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:28:52 smithi044 podman[98505]: 2024-09-17 07:28:52.998413992 +0000 UTC m=+0.632597261 container remove 7655e43d99443ab88f001c32cd5778554910f0bc7e2219421e4ac99ef5a5de5a (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2) 2024-09-17T07:28:53.643 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:28:53 smithi044 podman[98597]: 2024-09-17 07:28:53.2352377 +0000 UTC m=+0.016763187 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:28:53.643 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:28:53 smithi044 podman[98597]: 2024-09-17 07:28:53.349286644 +0000 UTC m=+0.130812184 container create b9c457fc8260f2acd1825612bd219f2e5b8a8dfb620f7cb95e98c871d358eb60 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1-deactivate, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-17T07:28:53.644 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:28:53 smithi044 podman[98597]: 2024-09-17 07:28:53.560165057 +0000 UTC m=+0.341690534 container init b9c457fc8260f2acd1825612bd219f2e5b8a8dfb620f7cb95e98c871d358eb60 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-17T07:28:53.644 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:28:53 smithi044 podman[98597]: 2024-09-17 07:28:53.563657723 +0000 UTC m=+0.345183206 container start b9c457fc8260f2acd1825612bd219f2e5b8a8dfb620f7cb95e98c871d358eb60 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True) 2024-09-17T07:28:53.644 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:28:53 smithi044 podman[98597]: 2024-09-17 07:28:53.643375165 +0000 UTC m=+0.424900651 container attach b9c457fc8260f2acd1825612bd219f2e5b8a8dfb620f7cb95e98c871d358eb60 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1-deactivate, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T07:28:54.074 INFO:journalctl@ceph.osd.1.smithi044.stdout:Sep 17 07:28:53 smithi044 podman[98597]: 2024-09-17 07:28:53.764277656 +0000 UTC m=+0.545803140 container died b9c457fc8260f2acd1825612bd219f2e5b8a8dfb620f7cb95e98c871d358eb60 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-1-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T07:28:54.370 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.1.service' 2024-09-17T07:28:54.414 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T07:28:54.415 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-17T07:28:54.415 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-17T07:28:54.415 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.2 2024-09-17T07:28:54.751 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:28:54 smithi044 systemd[1]: Stopping Ceph osd.2 for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:28:55.074 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:28:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2[71060]: 2024-09-17T07:28:54.750+0000 7f81b8daa640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T07:28:55.074 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:28:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2[71060]: 2024-09-17T07:28:54.750+0000 7f81b8daa640 -1 osd.2 136 *** Got signal Terminated *** 2024-09-17T07:28:55.074 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:28:54 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2[71060]: 2024-09-17T07:28:54.750+0000 7f81b8daa640 -1 osd.2 136 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T07:29:00.343 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:29:00 smithi044 podman[98701]: 2024-09-17 07:29:00.091272829 +0000 UTC m=+5.441788072 container died 43a48289eadad873c5d4b9da17d57c13225faa0b2e2f9a64fbbcbe7b851e8a83 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-17T07:29:00.634 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:29:00 smithi044 podman[98701]: 2024-09-17 07:29:00.44995763 +0000 UTC m=+5.800472905 container cleanup 43a48289eadad873c5d4b9da17d57c13225faa0b2e2f9a64fbbcbe7b851e8a83 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2) 2024-09-17T07:29:00.634 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:29:00 smithi044 bash[98701]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2 2024-09-17T07:29:01.074 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:29:00 smithi044 podman[98716]: 2024-09-17 07:29:00.634414096 +0000 UTC m=+0.536495904 container remove 43a48289eadad873c5d4b9da17d57c13225faa0b2e2f9a64fbbcbe7b851e8a83 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-17T07:29:01.405 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:29:01 smithi044 podman[98804]: 2024-09-17 07:29:00.981621731 +0000 UTC m=+0.016690079 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:29:01.405 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:29:01 smithi044 podman[98804]: 2024-09-17 07:29:01.085300598 +0000 UTC m=+0.120368933 container create adf652e504ea7a9c134635bcfded3654db801ec38797918c0af31a6e8466a47f (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T07:29:01.405 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:29:01 smithi044 podman[98804]: 2024-09-17 07:29:01.316496979 +0000 UTC m=+0.351565318 container init adf652e504ea7a9c134635bcfded3654db801ec38797918c0af31a6e8466a47f (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-17T07:29:01.405 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:29:01 smithi044 podman[98804]: 2024-09-17 07:29:01.320205676 +0000 UTC m=+0.355274005 container start adf652e504ea7a9c134635bcfded3654db801ec38797918c0af31a6e8466a47f (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-17T07:29:01.405 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:29:01 smithi044 podman[98804]: 2024-09-17 07:29:01.404468288 +0000 UTC m=+0.439536626 container attach adf652e504ea7a9c134635bcfded3654db801ec38797918c0af31a6e8466a47f (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-17T07:29:01.824 INFO:journalctl@ceph.osd.2.smithi044.stdout:Sep 17 07:29:01 smithi044 podman[98804]: 2024-09-17 07:29:01.515513178 +0000 UTC m=+0.550581514 container died adf652e504ea7a9c134635bcfded3654db801ec38797918c0af31a6e8466a47f (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-2-deactivate, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-17T07:29:02.162 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.2.service' 2024-09-17T07:29:02.206 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T07:29:02.206 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-17T07:29:02.207 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-17T07:29:02.207 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.3 2024-09-17T07:29:02.528 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:29:02 smithi044 systemd[1]: Stopping Ceph osd.3 for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:29:02.824 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:29:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3[73768]: 2024-09-17T07:29:02.527+0000 7fbfddf51640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T07:29:02.824 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:29:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3[73768]: 2024-09-17T07:29:02.527+0000 7fbfddf51640 -1 osd.3 136 *** Got signal Terminated *** 2024-09-17T07:29:02.824 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:29:02 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3[73768]: 2024-09-17T07:29:02.527+0000 7fbfddf51640 -1 osd.3 136 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T07:29:03.114 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:29:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:29:02.652Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.44:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.44:8765: connect: connection refused" 2024-09-17T07:29:03.114 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:29:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:29:02.655Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.44:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.44:8765: connect: connection refused" 2024-09-17T07:29:03.114 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:29:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:29:02.655Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.44:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.44:8765: connect: connection refused" 2024-09-17T07:29:03.114 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:29:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:29:02.657Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.44:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.44:8765: connect: connection refused" 2024-09-17T07:29:03.114 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:29:02 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:29:02.657Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.44:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.44:8765: connect: connection refused" 2024-09-17T07:29:04.824 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:29:04 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3[73768]: 2024-09-17T07:29:04.457+0000 7fbfd655d640 -1 osd.3 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:42.081234+0000 front 2024-09-17T07:28:42.081278+0000 (oldest deadline 2024-09-17T07:29:04.381373+0000) 2024-09-17T07:29:05.824 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:29:05 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3[73768]: 2024-09-17T07:29:05.495+0000 7fbfd655d640 -1 osd.3 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:42.081234+0000 front 2024-09-17T07:28:42.081278+0000 (oldest deadline 2024-09-17T07:29:04.381373+0000) 2024-09-17T07:29:05.864 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:05 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:05.516+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:40.873698+0000 front 2024-09-17T07:28:40.873874+0000 (oldest deadline 2024-09-17T07:29:04.973661+0000) 2024-09-17T07:29:06.614 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:29:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4[53687]: 2024-09-17T07:29:06.335+0000 7f1131c2d640 -1 osd.4 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:39.589351+0000 front 2024-09-17T07:28:39.589473+0000 (oldest deadline 2024-09-17T07:29:05.489254+0000) 2024-09-17T07:29:06.614 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:06.528+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:40.873698+0000 front 2024-09-17T07:28:40.873874+0000 (oldest deadline 2024-09-17T07:29:04.973661+0000) 2024-09-17T07:29:06.615 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:06.516+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:43.250753+0000 front 2024-09-17T07:28:43.250667+0000 (oldest deadline 2024-09-17T07:29:06.150665+0000) 2024-09-17T07:29:06.824 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:29:06 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3[73768]: 2024-09-17T07:29:06.520+0000 7fbfd655d640 -1 osd.3 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:42.081234+0000 front 2024-09-17T07:28:42.081278+0000 (oldest deadline 2024-09-17T07:29:04.381373+0000) 2024-09-17T07:29:07.347 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:06 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5[56041]: 2024-09-17T07:29:06.909+0000 7f5321081640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:42.476857+0000 front 2024-09-17T07:28:42.476869+0000 (oldest deadline 2024-09-17T07:29:06.576729+0000) 2024-09-17T07:29:07.614 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:29:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4[53687]: 2024-09-17T07:29:07.347+0000 7f1131c2d640 -1 osd.4 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:39.589351+0000 front 2024-09-17T07:28:39.589473+0000 (oldest deadline 2024-09-17T07:29:05.489254+0000) 2024-09-17T07:29:07.614 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:07.501+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:40.873698+0000 front 2024-09-17T07:28:40.873874+0000 (oldest deadline 2024-09-17T07:29:04.973661+0000) 2024-09-17T07:29:07.614 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:07.547+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:43.250753+0000 front 2024-09-17T07:28:43.250667+0000 (oldest deadline 2024-09-17T07:29:06.150665+0000) 2024-09-17T07:29:08.114 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:07 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5[56041]: 2024-09-17T07:29:07.925+0000 7f5321081640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:42.476857+0000 front 2024-09-17T07:28:42.476869+0000 (oldest deadline 2024-09-17T07:29:06.576729+0000) 2024-09-17T07:29:08.273 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:29:08 smithi044 podman[98907]: 2024-09-17 07:29:08.016300474 +0000 UTC m=+5.575767885 container died b4323c928f39eccf3759a07f1e9cd5db2584c879de4d46cfebb699fe9d041f16 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, ceph=True, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-17T07:29:08.545 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:29:08 smithi044 podman[98907]: 2024-09-17 07:29:08.369147623 +0000 UTC m=+5.928615031 container cleanup b4323c928f39eccf3759a07f1e9cd5db2584c879de4d46cfebb699fe9d041f16 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3, RELEASE=reef-19e2a2b, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T07:29:08.546 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:29:08 smithi044 bash[98907]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3 2024-09-17T07:29:08.824 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:29:08 smithi044 podman[98924]: 2024-09-17 07:29:08.545556421 +0000 UTC m=+0.518832211 container remove b4323c928f39eccf3759a07f1e9cd5db2584c879de4d46cfebb699fe9d041f16 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=reef-19e2a2b, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T07:29:08.864 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:29:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4[53687]: 2024-09-17T07:29:08.369+0000 7f1131c2d640 -1 osd.4 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:39.589351+0000 front 2024-09-17T07:28:39.589473+0000 (oldest deadline 2024-09-17T07:29:05.489254+0000) 2024-09-17T07:29:08.864 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:08.476+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:40.873698+0000 front 2024-09-17T07:28:40.873874+0000 (oldest deadline 2024-09-17T07:29:04.973661+0000) 2024-09-17T07:29:08.865 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:08.539+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:43.250753+0000 front 2024-09-17T07:28:43.250667+0000 (oldest deadline 2024-09-17T07:29:06.150665+0000) 2024-09-17T07:29:09.282 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:29:08 smithi044 podman[99015]: 2024-09-17 07:29:08.900390853 +0000 UTC m=+0.016418597 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:29:09.282 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:29:09 smithi044 podman[99015]: 2024-09-17 07:29:09.013263511 +0000 UTC m=+0.129291238 container create f07e3bb00cf5774de1f1f870c5bed1d6bb7119673cbfed19f01434ff75aba0e9 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3-deactivate, RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-17T07:29:09.282 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:29:09 smithi044 podman[99015]: 2024-09-17 07:29:09.282021717 +0000 UTC m=+0.398049461 container init f07e3bb00cf5774de1f1f870c5bed1d6bb7119673cbfed19f01434ff75aba0e9 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-17T07:29:09.364 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:08 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5[56041]: 2024-09-17T07:29:08.918+0000 7f5321081640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:42.476857+0000 front 2024-09-17T07:28:42.476869+0000 (oldest deadline 2024-09-17T07:29:06.576729+0000) 2024-09-17T07:29:09.574 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:29:09 smithi044 podman[99015]: 2024-09-17 07:29:09.285646228 +0000 UTC m=+0.401673961 container start f07e3bb00cf5774de1f1f870c5bed1d6bb7119673cbfed19f01434ff75aba0e9 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-17T07:29:09.575 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:29:09 smithi044 podman[99015]: 2024-09-17 07:29:09.373895872 +0000 UTC m=+0.489923621 container attach f07e3bb00cf5774de1f1f870c5bed1d6bb7119673cbfed19f01434ff75aba0e9 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b) 2024-09-17T07:29:09.575 INFO:journalctl@ceph.osd.3.smithi044.stdout:Sep 17 07:29:09 smithi044 podman[99015]: 2024-09-17 07:29:09.475600471 +0000 UTC m=+0.591628205 container died f07e3bb00cf5774de1f1f870c5bed1d6bb7119673cbfed19f01434ff75aba0e9 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T07:29:09.864 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:29:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4[53687]: 2024-09-17T07:29:09.380+0000 7f1131c2d640 -1 osd.4 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:39.589351+0000 front 2024-09-17T07:28:39.589473+0000 (oldest deadline 2024-09-17T07:29:05.489254+0000) 2024-09-17T07:29:09.864 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:09.511+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:40.873698+0000 front 2024-09-17T07:28:40.873874+0000 (oldest deadline 2024-09-17T07:29:04.973661+0000) 2024-09-17T07:29:09.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:09.517+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:43.250753+0000 front 2024-09-17T07:28:43.250667+0000 (oldest deadline 2024-09-17T07:29:06.150665+0000) 2024-09-17T07:29:10.060 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.3.service' 2024-09-17T07:29:10.105 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T07:29:10.106 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-17T07:29:10.106 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-17T07:29:10.106 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.4 2024-09-17T07:29:10.162 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:09 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5[56041]: 2024-09-17T07:29:09.893+0000 7f5321081640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:42.476857+0000 front 2024-09-17T07:28:42.476869+0000 (oldest deadline 2024-09-17T07:29:06.576729+0000) 2024-09-17T07:29:10.522 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:29:10 smithi179 systemd[1]: Stopping Ceph osd.4 for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:29:10.522 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:29:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4[53687]: 2024-09-17T07:29:10.395+0000 7f1135619640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T07:29:10.522 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:29:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4[53687]: 2024-09-17T07:29:10.395+0000 7f1135619640 -1 osd.4 136 *** Got signal Terminated *** 2024-09-17T07:29:10.522 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:29:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4[53687]: 2024-09-17T07:29:10.395+0000 7f1135619640 -1 osd.4 136 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T07:29:10.522 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:29:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4[53687]: 2024-09-17T07:29:10.408+0000 7f1131c2d640 -1 osd.4 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:39.589351+0000 front 2024-09-17T07:28:39.589473+0000 (oldest deadline 2024-09-17T07:29:05.489254+0000) 2024-09-17T07:29:10.860 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:10.521+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:40.873698+0000 front 2024-09-17T07:28:40.873874+0000 (oldest deadline 2024-09-17T07:29:04.973661+0000) 2024-09-17T07:29:10.861 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:10.518+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:43.250753+0000 front 2024-09-17T07:28:43.250667+0000 (oldest deadline 2024-09-17T07:29:06.150665+0000) 2024-09-17T07:29:11.113 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:10 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5[56041]: 2024-09-17T07:29:10.860+0000 7f5321081640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:42.476857+0000 front 2024-09-17T07:28:42.476869+0000 (oldest deadline 2024-09-17T07:29:06.576729+0000) 2024-09-17T07:29:11.864 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:11 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:11.480+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:40.873698+0000 front 2024-09-17T07:28:40.873874+0000 (oldest deadline 2024-09-17T07:29:04.973661+0000) 2024-09-17T07:29:11.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:11 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:11.499+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:43.250753+0000 front 2024-09-17T07:28:43.250667+0000 (oldest deadline 2024-09-17T07:29:06.150665+0000) 2024-09-17T07:29:11.864 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:29:11 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4[53687]: 2024-09-17T07:29:11.411+0000 7f1131c2d640 -1 osd.4 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:39.589351+0000 front 2024-09-17T07:28:39.589473+0000 (oldest deadline 2024-09-17T07:29:05.489254+0000) 2024-09-17T07:29:12.364 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:11 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5[56041]: 2024-09-17T07:29:11.891+0000 7f5321081640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:42.476857+0000 front 2024-09-17T07:28:42.476869+0000 (oldest deadline 2024-09-17T07:29:06.576729+0000) 2024-09-17T07:29:12.852 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:29:12 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4[53687]: 2024-09-17T07:29:12.366+0000 7f1131c2d640 -1 osd.4 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:39.589351+0000 front 2024-09-17T07:28:39.589473+0000 (oldest deadline 2024-09-17T07:29:05.489254+0000) 2024-09-17T07:29:12.852 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:12 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:12.468+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:40.873698+0000 front 2024-09-17T07:28:40.873874+0000 (oldest deadline 2024-09-17T07:29:04.973661+0000) 2024-09-17T07:29:12.852 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:12 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:12.471+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:43.250753+0000 front 2024-09-17T07:28:43.250667+0000 (oldest deadline 2024-09-17T07:29:06.150665+0000) 2024-09-17T07:29:13.114 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:12 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5[56041]: 2024-09-17T07:29:12.851+0000 7f5321081640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:42.476857+0000 front 2024-09-17T07:28:42.476869+0000 (oldest deadline 2024-09-17T07:29:06.576729+0000) 2024-09-17T07:29:13.614 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:29:13 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4[53687]: 2024-09-17T07:29:13.325+0000 7f1131c2d640 -1 osd.4 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:39.589351+0000 front 2024-09-17T07:28:39.589473+0000 (oldest deadline 2024-09-17T07:29:05.489254+0000) 2024-09-17T07:29:13.614 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:13 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:13.496+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:40.873698+0000 front 2024-09-17T07:28:40.873874+0000 (oldest deadline 2024-09-17T07:29:04.973661+0000) 2024-09-17T07:29:13.614 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:13 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:13.472+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:43.250753+0000 front 2024-09-17T07:28:43.250667+0000 (oldest deadline 2024-09-17T07:29:06.150665+0000) 2024-09-17T07:29:14.114 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:13 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5[56041]: 2024-09-17T07:29:13.858+0000 7f5321081640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:42.476857+0000 front 2024-09-17T07:28:42.476869+0000 (oldest deadline 2024-09-17T07:29:06.576729+0000) 2024-09-17T07:29:14.614 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:29:14 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4[53687]: 2024-09-17T07:29:14.327+0000 7f1131c2d640 -1 osd.4 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:39.589351+0000 front 2024-09-17T07:28:39.589473+0000 (oldest deadline 2024-09-17T07:29:05.489254+0000) 2024-09-17T07:29:14.614 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:14 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:14.474+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:43.250753+0000 front 2024-09-17T07:28:43.250667+0000 (oldest deadline 2024-09-17T07:29:06.150665+0000) 2024-09-17T07:29:14.614 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:14 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:14.485+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:40.873698+0000 front 2024-09-17T07:28:40.873874+0000 (oldest deadline 2024-09-17T07:29:04.973661+0000) 2024-09-17T07:29:14.614 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:14 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:14.485+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:49.575121+0000 front 2024-09-17T07:28:49.575207+0000 (oldest deadline 2024-09-17T07:29:14.275149+0000) 2024-09-17T07:29:15.114 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:14 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5[56041]: 2024-09-17T07:29:14.855+0000 7f5321081640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:42.476857+0000 front 2024-09-17T07:28:42.476869+0000 (oldest deadline 2024-09-17T07:29:06.576729+0000) 2024-09-17T07:29:15.765 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:29:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4[53687]: 2024-09-17T07:29:15.369+0000 7f1131c2d640 -1 osd.4 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:39.589351+0000 front 2024-09-17T07:28:39.589473+0000 (oldest deadline 2024-09-17T07:29:05.489254+0000) 2024-09-17T07:29:15.765 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:29:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4[53687]: 2024-09-17T07:29:15.369+0000 7f1131c2d640 -1 osd.4 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:51.091564+0000 front 2024-09-17T07:28:51.091651+0000 (oldest deadline 2024-09-17T07:29:15.191424+0000) 2024-09-17T07:29:15.765 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:15.427+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:43.250753+0000 front 2024-09-17T07:28:43.250667+0000 (oldest deadline 2024-09-17T07:29:06.150665+0000) 2024-09-17T07:29:15.765 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:15.486+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:40.873698+0000 front 2024-09-17T07:28:40.873874+0000 (oldest deadline 2024-09-17T07:29:04.973661+0000) 2024-09-17T07:29:15.765 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:15.486+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:49.575121+0000 front 2024-09-17T07:28:49.575207+0000 (oldest deadline 2024-09-17T07:29:14.275149+0000) 2024-09-17T07:29:16.055 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:29:15 smithi179 podman[80143]: 2024-09-17 07:29:15.764556315 +0000 UTC m=+5.453726336 container died 917adc8feb0bbbd3f9514ac1b01c37bba11ecb0d72d70fa6e7e99db8977404d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-17T07:29:16.055 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:15 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5[56041]: 2024-09-17T07:29:15.869+0000 7f5321081640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:42.476857+0000 front 2024-09-17T07:28:42.476869+0000 (oldest deadline 2024-09-17T07:29:06.576729+0000) 2024-09-17T07:29:16.364 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:29:16 smithi179 podman[80143]: 2024-09-17 07:29:16.23757817 +0000 UTC m=+5.926748172 container cleanup 917adc8feb0bbbd3f9514ac1b01c37bba11ecb0d72d70fa6e7e99db8977404d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T07:29:16.364 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:29:16 smithi179 bash[80143]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4 2024-09-17T07:29:16.848 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:29:16 smithi179 podman[80158]: 2024-09-17 07:29:16.556180511 +0000 UTC m=+0.784439220 container remove 917adc8feb0bbbd3f9514ac1b01c37bba11ecb0d72d70fa6e7e99db8977404d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-17T07:29:16.849 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:16 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:16.515+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:40.873698+0000 front 2024-09-17T07:28:40.873874+0000 (oldest deadline 2024-09-17T07:29:04.973661+0000) 2024-09-17T07:29:16.849 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:16 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:16.515+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:49.575121+0000 front 2024-09-17T07:28:49.575207+0000 (oldest deadline 2024-09-17T07:29:14.275149+0000) 2024-09-17T07:29:16.849 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:16 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:16.460+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:43.250753+0000 front 2024-09-17T07:28:43.250667+0000 (oldest deadline 2024-09-17T07:29:06.150665+0000) 2024-09-17T07:29:17.114 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:16 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5[56041]: 2024-09-17T07:29:16.847+0000 7f5321081640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:42.476857+0000 front 2024-09-17T07:28:42.476869+0000 (oldest deadline 2024-09-17T07:29:06.576729+0000) 2024-09-17T07:29:17.114 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:29:16 smithi179 podman[80249]: 2024-09-17 07:29:16.818104927 +0000 UTC m=+0.016222824 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:29:17.114 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:29:16 smithi179 podman[80249]: 2024-09-17 07:29:16.948615447 +0000 UTC m=+0.146733346 container create 4f6a2d060e710c1994a1472e91f9f5f50f312318b86ddaa4d2b42023b420a176 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-17T07:29:17.377 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:29:17 smithi179 podman[80249]: 2024-09-17 07:29:17.193033775 +0000 UTC m=+0.391151685 container init 4f6a2d060e710c1994a1472e91f9f5f50f312318b86ddaa4d2b42023b420a176 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-17T07:29:17.377 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:29:17 smithi179 podman[80249]: 2024-09-17 07:29:17.196786377 +0000 UTC m=+0.394904274 container start 4f6a2d060e710c1994a1472e91f9f5f50f312318b86ddaa4d2b42023b420a176 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-17T07:29:17.378 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:29:17 smithi179 podman[80249]: 2024-09-17 07:29:17.292959377 +0000 UTC m=+0.491077292 container attach 4f6a2d060e710c1994a1472e91f9f5f50f312318b86ddaa4d2b42023b420a176 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b) 2024-09-17T07:29:17.378 INFO:journalctl@ceph.osd.4.smithi179.stdout:Sep 17 07:29:17 smithi179 podman[80249]: 2024-09-17 07:29:17.377545063 +0000 UTC m=+0.575662960 container died 4f6a2d060e710c1994a1472e91f9f5f50f312318b86ddaa4d2b42023b420a176 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-17T07:29:17.738 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:17.509+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:40.873698+0000 front 2024-09-17T07:28:40.873874+0000 (oldest deadline 2024-09-17T07:29:04.973661+0000) 2024-09-17T07:29:17.738 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:17.509+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:49.575121+0000 front 2024-09-17T07:28:49.575207+0000 (oldest deadline 2024-09-17T07:29:14.275149+0000) 2024-09-17T07:29:17.738 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:17.463+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:43.250753+0000 front 2024-09-17T07:28:43.250667+0000 (oldest deadline 2024-09-17T07:29:06.150665+0000) 2024-09-17T07:29:17.738 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:17.463+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:51.352234+0000 front 2024-09-17T07:28:51.352171+0000 (oldest deadline 2024-09-17T07:29:17.252046+0000) 2024-09-17T07:29:18.114 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5[56041]: 2024-09-17T07:29:17.846+0000 7f5321081640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:42.476857+0000 front 2024-09-17T07:28:42.476869+0000 (oldest deadline 2024-09-17T07:29:06.576729+0000) 2024-09-17T07:29:18.114 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:17 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5[56041]: 2024-09-17T07:29:17.846+0000 7f5321081640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:46.577240+0000 front 2024-09-17T07:28:46.577249+0000 (oldest deadline 2024-09-17T07:29:17.777794+0000) 2024-09-17T07:29:18.176 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.4.service' 2024-09-17T07:29:18.221 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T07:29:18.221 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-17T07:29:18.221 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-17T07:29:18.222 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.5 2024-09-17T07:29:18.462 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:18 smithi179 systemd[1]: Stopping Ceph osd.5 for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:29:18.852 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:18 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:18.461+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:43.250753+0000 front 2024-09-17T07:28:43.250667+0000 (oldest deadline 2024-09-17T07:29:06.150665+0000) 2024-09-17T07:29:18.852 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:18 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:18.461+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:51.352234+0000 front 2024-09-17T07:28:51.352171+0000 (oldest deadline 2024-09-17T07:29:17.252046+0000) 2024-09-17T07:29:18.853 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:18 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5[56041]: 2024-09-17T07:29:18.550+0000 7f5324a6d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T07:29:18.853 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:18 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5[56041]: 2024-09-17T07:29:18.550+0000 7f5324a6d640 -1 osd.5 136 *** Got signal Terminated *** 2024-09-17T07:29:18.853 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:18 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5[56041]: 2024-09-17T07:29:18.550+0000 7f5324a6d640 -1 osd.5 136 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T07:29:18.853 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:18 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:18.537+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:40.873698+0000 front 2024-09-17T07:28:40.873874+0000 (oldest deadline 2024-09-17T07:29:04.973661+0000) 2024-09-17T07:29:18.853 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:18 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:18.537+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:49.575121+0000 front 2024-09-17T07:28:49.575207+0000 (oldest deadline 2024-09-17T07:29:14.275149+0000) 2024-09-17T07:29:19.114 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:18 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5[56041]: 2024-09-17T07:29:18.851+0000 7f5321081640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:42.476857+0000 front 2024-09-17T07:28:42.476869+0000 (oldest deadline 2024-09-17T07:29:06.576729+0000) 2024-09-17T07:29:19.114 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:18 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5[56041]: 2024-09-17T07:29:18.851+0000 7f5321081640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:46.577240+0000 front 2024-09-17T07:28:46.577249+0000 (oldest deadline 2024-09-17T07:29:17.777794+0000) 2024-09-17T07:29:19.806 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:19 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:19.499+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:43.250753+0000 front 2024-09-17T07:28:43.250667+0000 (oldest deadline 2024-09-17T07:29:06.150665+0000) 2024-09-17T07:29:19.807 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:19 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:19.499+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:51.352234+0000 front 2024-09-17T07:28:51.352171+0000 (oldest deadline 2024-09-17T07:29:17.252046+0000) 2024-09-17T07:29:19.807 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:19 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:19.583+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:40.873698+0000 front 2024-09-17T07:28:40.873874+0000 (oldest deadline 2024-09-17T07:29:04.973661+0000) 2024-09-17T07:29:19.807 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:19 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:19.583+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:49.575121+0000 front 2024-09-17T07:28:49.575207+0000 (oldest deadline 2024-09-17T07:29:14.275149+0000) 2024-09-17T07:29:20.117 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:19 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5[56041]: 2024-09-17T07:29:19.806+0000 7f5321081640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:42.476857+0000 front 2024-09-17T07:28:42.476869+0000 (oldest deadline 2024-09-17T07:29:06.576729+0000) 2024-09-17T07:29:20.117 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:19 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5[56041]: 2024-09-17T07:29:19.806+0000 7f5321081640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:46.577240+0000 front 2024-09-17T07:28:46.577249+0000 (oldest deadline 2024-09-17T07:29:17.777794+0000) 2024-09-17T07:29:20.763 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:20.466+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:43.250753+0000 front 2024-09-17T07:28:43.250667+0000 (oldest deadline 2024-09-17T07:29:06.150665+0000) 2024-09-17T07:29:20.763 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:20.466+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:51.352234+0000 front 2024-09-17T07:28:51.352171+0000 (oldest deadline 2024-09-17T07:29:17.252046+0000) 2024-09-17T07:29:20.763 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:20.630+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:40.873698+0000 front 2024-09-17T07:28:40.873874+0000 (oldest deadline 2024-09-17T07:29:04.973661+0000) 2024-09-17T07:29:20.763 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:20.630+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:49.575121+0000 front 2024-09-17T07:28:49.575207+0000 (oldest deadline 2024-09-17T07:29:14.275149+0000) 2024-09-17T07:29:20.763 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:20.630+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6822 osd.2 since back 2024-09-17T07:28:58.876398+0000 front 2024-09-17T07:28:58.876370+0000 (oldest deadline 2024-09-17T07:29:20.576422+0000) 2024-09-17T07:29:21.114 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5[56041]: 2024-09-17T07:29:20.762+0000 7f5321081640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:42.476857+0000 front 2024-09-17T07:28:42.476869+0000 (oldest deadline 2024-09-17T07:29:06.576729+0000) 2024-09-17T07:29:21.114 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:20 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5[56041]: 2024-09-17T07:29:20.762+0000 7f5321081640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:46.577240+0000 front 2024-09-17T07:28:46.577249+0000 (oldest deadline 2024-09-17T07:29:17.777794+0000) 2024-09-17T07:29:21.864 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:21 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5[56041]: 2024-09-17T07:29:21.737+0000 7f5321081640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:42.476857+0000 front 2024-09-17T07:28:42.476869+0000 (oldest deadline 2024-09-17T07:29:06.576729+0000) 2024-09-17T07:29:21.864 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:21 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5[56041]: 2024-09-17T07:29:21.737+0000 7f5321081640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:46.577240+0000 front 2024-09-17T07:28:46.577249+0000 (oldest deadline 2024-09-17T07:29:17.777794+0000) 2024-09-17T07:29:21.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:21 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:21.495+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:43.250753+0000 front 2024-09-17T07:28:43.250667+0000 (oldest deadline 2024-09-17T07:29:06.150665+0000) 2024-09-17T07:29:21.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:21 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:21.495+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:51.352234+0000 front 2024-09-17T07:28:51.352171+0000 (oldest deadline 2024-09-17T07:29:17.252046+0000) 2024-09-17T07:29:21.865 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:21 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:21.609+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:40.873698+0000 front 2024-09-17T07:28:40.873874+0000 (oldest deadline 2024-09-17T07:29:04.973661+0000) 2024-09-17T07:29:21.865 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:21 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:21.609+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:49.575121+0000 front 2024-09-17T07:28:49.575207+0000 (oldest deadline 2024-09-17T07:29:14.275149+0000) 2024-09-17T07:29:21.865 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:21 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:21.609+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6822 osd.2 since back 2024-09-17T07:28:58.876398+0000 front 2024-09-17T07:28:58.876370+0000 (oldest deadline 2024-09-17T07:29:20.576422+0000) 2024-09-17T07:29:22.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:22.455+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:43.250753+0000 front 2024-09-17T07:28:43.250667+0000 (oldest deadline 2024-09-17T07:29:06.150665+0000) 2024-09-17T07:29:22.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:22.455+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:51.352234+0000 front 2024-09-17T07:28:51.352171+0000 (oldest deadline 2024-09-17T07:29:17.252046+0000) 2024-09-17T07:29:22.864 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5[56041]: 2024-09-17T07:29:22.696+0000 7f5321081640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:42.476857+0000 front 2024-09-17T07:28:42.476869+0000 (oldest deadline 2024-09-17T07:29:06.576729+0000) 2024-09-17T07:29:22.864 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5[56041]: 2024-09-17T07:29:22.696+0000 7f5321081640 -1 osd.5 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:46.577240+0000 front 2024-09-17T07:28:46.577249+0000 (oldest deadline 2024-09-17T07:29:17.777794+0000) 2024-09-17T07:29:22.865 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:22.585+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:40.873698+0000 front 2024-09-17T07:28:40.873874+0000 (oldest deadline 2024-09-17T07:29:04.973661+0000) 2024-09-17T07:29:22.865 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:22.585+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:49.575121+0000 front 2024-09-17T07:28:49.575207+0000 (oldest deadline 2024-09-17T07:29:14.275149+0000) 2024-09-17T07:29:22.865 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:22 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:22.585+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6822 osd.2 since back 2024-09-17T07:28:58.876398+0000 front 2024-09-17T07:28:58.876370+0000 (oldest deadline 2024-09-17T07:29:20.576422+0000) 2024-09-17T07:29:23.864 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:23.581+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:40.873698+0000 front 2024-09-17T07:28:40.873874+0000 (oldest deadline 2024-09-17T07:29:04.973661+0000) 2024-09-17T07:29:23.864 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:23.581+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:49.575121+0000 front 2024-09-17T07:28:49.575207+0000 (oldest deadline 2024-09-17T07:29:14.275149+0000) 2024-09-17T07:29:23.864 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:23.581+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6822 osd.2 since back 2024-09-17T07:28:58.876398+0000 front 2024-09-17T07:28:58.876370+0000 (oldest deadline 2024-09-17T07:29:20.576422+0000) 2024-09-17T07:29:23.865 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:23.498+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:43.250753+0000 front 2024-09-17T07:28:43.250667+0000 (oldest deadline 2024-09-17T07:29:06.150665+0000) 2024-09-17T07:29:23.865 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:23.498+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:51.352234+0000 front 2024-09-17T07:28:51.352171+0000 (oldest deadline 2024-09-17T07:29:17.252046+0000) 2024-09-17T07:29:23.865 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:23 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:23.498+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6822 osd.2 since back 2024-09-17T07:28:57.753273+0000 front 2024-09-17T07:28:57.753163+0000 (oldest deadline 2024-09-17T07:29:23.053104+0000) 2024-09-17T07:29:24.186 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:23 smithi179 podman[80352]: 2024-09-17 07:29:23.900395428 +0000 UTC m=+5.452454928 container died 0258eb8ed0e71089ea84a1288783e16a1661df4b2d29a05838957a419f38dc34 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-17T07:29:24.475 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:24 smithi179 podman[80352]: 2024-09-17 07:29:24.375558565 +0000 UTC m=+5.927618056 container cleanup 0258eb8ed0e71089ea84a1288783e16a1661df4b2d29a05838957a419f38dc34 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, ceph=True) 2024-09-17T07:29:24.475 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:24 smithi179 bash[80352]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5 2024-09-17T07:29:24.864 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:24 smithi179 podman[80366]: 2024-09-17 07:29:24.643992137 +0000 UTC m=+0.736902386 container remove 0258eb8ed0e71089ea84a1288783e16a1661df4b2d29a05838957a419f38dc34 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T07:29:24.864 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:24 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:24.604+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:40.873698+0000 front 2024-09-17T07:28:40.873874+0000 (oldest deadline 2024-09-17T07:29:04.973661+0000) 2024-09-17T07:29:24.864 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:24 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:24.604+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:49.575121+0000 front 2024-09-17T07:28:49.575207+0000 (oldest deadline 2024-09-17T07:29:14.275149+0000) 2024-09-17T07:29:24.864 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:24 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:24.604+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6822 osd.2 since back 2024-09-17T07:28:58.876398+0000 front 2024-09-17T07:28:58.876370+0000 (oldest deadline 2024-09-17T07:29:20.576422+0000) 2024-09-17T07:29:24.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:24 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:24.474+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:43.250753+0000 front 2024-09-17T07:28:43.250667+0000 (oldest deadline 2024-09-17T07:29:06.150665+0000) 2024-09-17T07:29:24.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:24 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:24.474+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:51.352234+0000 front 2024-09-17T07:28:51.352171+0000 (oldest deadline 2024-09-17T07:29:17.252046+0000) 2024-09-17T07:29:24.865 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:24 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:24.474+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6822 osd.2 since back 2024-09-17T07:28:57.753273+0000 front 2024-09-17T07:28:57.753163+0000 (oldest deadline 2024-09-17T07:29:23.053104+0000) 2024-09-17T07:29:25.255 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:24 smithi179 podman[80459]: 2024-09-17 07:29:24.900689673 +0000 UTC m=+0.016188150 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:29:25.255 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:25 smithi179 podman[80459]: 2024-09-17 07:29:25.019653052 +0000 UTC m=+0.135151529 container create b7d9d53cec287cdf0e5aef2de8cc864e255d46ebcff210a44082128e90991db2 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS) 2024-09-17T07:29:25.255 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:25 smithi179 podman[80459]: 2024-09-17 07:29:25.255335545 +0000 UTC m=+0.370834028 container init b7d9d53cec287cdf0e5aef2de8cc864e255d46ebcff210a44082128e90991db2 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5-deactivate, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731) 2024-09-17T07:29:25.523 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:25 smithi179 podman[80459]: 2024-09-17 07:29:25.258629503 +0000 UTC m=+0.374127984 container start b7d9d53cec287cdf0e5aef2de8cc864e255d46ebcff210a44082128e90991db2 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T07:29:25.523 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:25 smithi179 podman[80459]: 2024-09-17 07:29:25.347033068 +0000 UTC m=+0.462531547 container attach b7d9d53cec287cdf0e5aef2de8cc864e255d46ebcff210a44082128e90991db2 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True) 2024-09-17T07:29:25.523 INFO:journalctl@ceph.osd.5.smithi179.stdout:Sep 17 07:29:25 smithi179 podman[80459]: 2024-09-17 07:29:25.455057177 +0000 UTC m=+0.570555661 container died b7d9d53cec287cdf0e5aef2de8cc864e255d46ebcff210a44082128e90991db2 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-17T07:29:25.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:25 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:25.521+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:43.250753+0000 front 2024-09-17T07:28:43.250667+0000 (oldest deadline 2024-09-17T07:29:06.150665+0000) 2024-09-17T07:29:25.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:25 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:25.521+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:51.352234+0000 front 2024-09-17T07:28:51.352171+0000 (oldest deadline 2024-09-17T07:29:17.252046+0000) 2024-09-17T07:29:25.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:25 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:25.521+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6822 osd.2 since back 2024-09-17T07:28:57.753273+0000 front 2024-09-17T07:28:57.753163+0000 (oldest deadline 2024-09-17T07:29:23.053104+0000) 2024-09-17T07:29:25.864 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:25 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:25.612+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:40.873698+0000 front 2024-09-17T07:28:40.873874+0000 (oldest deadline 2024-09-17T07:29:04.973661+0000) 2024-09-17T07:29:25.865 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:25 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:25.612+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:49.575121+0000 front 2024-09-17T07:28:49.575207+0000 (oldest deadline 2024-09-17T07:29:14.275149+0000) 2024-09-17T07:29:25.865 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:25 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:25.612+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6822 osd.2 since back 2024-09-17T07:28:58.876398+0000 front 2024-09-17T07:28:58.876370+0000 (oldest deadline 2024-09-17T07:29:20.576422+0000) 2024-09-17T07:29:26.034 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.5.service' 2024-09-17T07:29:26.074 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T07:29:26.074 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-17T07:29:26.074 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-17T07:29:26.075 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.6 2024-09-17T07:29:26.364 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:26 smithi179 systemd[1]: Stopping Ceph osd.6 for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:29:26.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:26 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:26.532+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:43.250753+0000 front 2024-09-17T07:28:43.250667+0000 (oldest deadline 2024-09-17T07:29:06.150665+0000) 2024-09-17T07:29:26.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:26 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:26.532+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:51.352234+0000 front 2024-09-17T07:28:51.352171+0000 (oldest deadline 2024-09-17T07:29:17.252046+0000) 2024-09-17T07:29:26.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:26 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:26.532+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6822 osd.2 since back 2024-09-17T07:28:57.753273+0000 front 2024-09-17T07:28:57.753163+0000 (oldest deadline 2024-09-17T07:29:23.053104+0000) 2024-09-17T07:29:26.864 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:26 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:26.394+0000 7f2ff9ab8640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T07:29:26.864 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:26 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:26.394+0000 7f2ff9ab8640 -1 osd.6 136 *** Got signal Terminated *** 2024-09-17T07:29:26.865 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:26 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:26.394+0000 7f2ff9ab8640 -1 osd.6 136 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T07:29:26.865 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:26 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:26.612+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:40.873698+0000 front 2024-09-17T07:28:40.873874+0000 (oldest deadline 2024-09-17T07:29:04.973661+0000) 2024-09-17T07:29:26.865 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:26 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:26.612+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:49.575121+0000 front 2024-09-17T07:28:49.575207+0000 (oldest deadline 2024-09-17T07:29:14.275149+0000) 2024-09-17T07:29:26.865 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:26 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:26.612+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6822 osd.2 since back 2024-09-17T07:28:58.876398+0000 front 2024-09-17T07:28:58.876370+0000 (oldest deadline 2024-09-17T07:29:20.576422+0000) 2024-09-17T07:29:27.864 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:27 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:27.631+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:40.873698+0000 front 2024-09-17T07:28:40.873874+0000 (oldest deadline 2024-09-17T07:29:04.973661+0000) 2024-09-17T07:29:27.864 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:27 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:27.631+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:49.575121+0000 front 2024-09-17T07:28:49.575207+0000 (oldest deadline 2024-09-17T07:29:14.275149+0000) 2024-09-17T07:29:27.864 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:27 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:27.631+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6822 osd.2 since back 2024-09-17T07:28:58.876398+0000 front 2024-09-17T07:28:58.876370+0000 (oldest deadline 2024-09-17T07:29:20.576422+0000) 2024-09-17T07:29:27.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:27 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:27.518+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:43.250753+0000 front 2024-09-17T07:28:43.250667+0000 (oldest deadline 2024-09-17T07:29:06.150665+0000) 2024-09-17T07:29:27.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:27 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:27.518+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:51.352234+0000 front 2024-09-17T07:28:51.352171+0000 (oldest deadline 2024-09-17T07:29:17.252046+0000) 2024-09-17T07:29:27.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:27 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:27.518+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6822 osd.2 since back 2024-09-17T07:28:57.753273+0000 front 2024-09-17T07:28:57.753163+0000 (oldest deadline 2024-09-17T07:29:23.053104+0000) 2024-09-17T07:29:28.864 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:28 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:28.582+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:40.873698+0000 front 2024-09-17T07:28:40.873874+0000 (oldest deadline 2024-09-17T07:29:04.973661+0000) 2024-09-17T07:29:28.864 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:28 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:28.582+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:49.575121+0000 front 2024-09-17T07:28:49.575207+0000 (oldest deadline 2024-09-17T07:29:14.275149+0000) 2024-09-17T07:29:28.864 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:28 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:28.582+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6822 osd.2 since back 2024-09-17T07:28:58.876398+0000 front 2024-09-17T07:28:58.876370+0000 (oldest deadline 2024-09-17T07:29:20.576422+0000) 2024-09-17T07:29:28.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:28 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:28.476+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:43.250753+0000 front 2024-09-17T07:28:43.250667+0000 (oldest deadline 2024-09-17T07:29:06.150665+0000) 2024-09-17T07:29:28.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:28 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:28.476+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:51.352234+0000 front 2024-09-17T07:28:51.352171+0000 (oldest deadline 2024-09-17T07:29:17.252046+0000) 2024-09-17T07:29:28.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:28 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:28.476+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6822 osd.2 since back 2024-09-17T07:28:57.753273+0000 front 2024-09-17T07:28:57.753163+0000 (oldest deadline 2024-09-17T07:29:23.053104+0000) 2024-09-17T07:29:29.864 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:29.580+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:40.873698+0000 front 2024-09-17T07:28:40.873874+0000 (oldest deadline 2024-09-17T07:29:04.973661+0000) 2024-09-17T07:29:29.864 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:29.580+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:49.575121+0000 front 2024-09-17T07:28:49.575207+0000 (oldest deadline 2024-09-17T07:29:14.275149+0000) 2024-09-17T07:29:29.864 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:29.580+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6822 osd.2 since back 2024-09-17T07:28:58.876398+0000 front 2024-09-17T07:28:58.876370+0000 (oldest deadline 2024-09-17T07:29:20.576422+0000) 2024-09-17T07:29:29.864 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:29.580+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6830 osd.3 since back 2024-09-17T07:29:04.077426+0000 front 2024-09-17T07:29:04.077361+0000 (oldest deadline 2024-09-17T07:29:28.777334+0000) 2024-09-17T07:29:29.865 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:29.496+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:43.250753+0000 front 2024-09-17T07:28:43.250667+0000 (oldest deadline 2024-09-17T07:29:06.150665+0000) 2024-09-17T07:29:29.865 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:29.496+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:51.352234+0000 front 2024-09-17T07:28:51.352171+0000 (oldest deadline 2024-09-17T07:29:17.252046+0000) 2024-09-17T07:29:29.865 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:29.496+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6822 osd.2 since back 2024-09-17T07:28:57.753273+0000 front 2024-09-17T07:28:57.753163+0000 (oldest deadline 2024-09-17T07:29:23.053104+0000) 2024-09-17T07:29:29.865 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:29 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:29.496+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6830 osd.3 since back 2024-09-17T07:29:04.754248+0000 front 2024-09-17T07:29:04.754234+0000 (oldest deadline 2024-09-17T07:29:29.454187+0000) 2024-09-17T07:29:30.864 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:30 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:30.580+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:40.873698+0000 front 2024-09-17T07:28:40.873874+0000 (oldest deadline 2024-09-17T07:29:04.973661+0000) 2024-09-17T07:29:30.864 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:30 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:30.580+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:49.575121+0000 front 2024-09-17T07:28:49.575207+0000 (oldest deadline 2024-09-17T07:29:14.275149+0000) 2024-09-17T07:29:30.864 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:30 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:30.580+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6822 osd.2 since back 2024-09-17T07:28:58.876398+0000 front 2024-09-17T07:28:58.876370+0000 (oldest deadline 2024-09-17T07:29:20.576422+0000) 2024-09-17T07:29:30.864 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:30 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6[58374]: 2024-09-17T07:29:30.580+0000 7f2ff60cc640 -1 osd.6 136 heartbeat_check: no reply from 172.21.15.44:6830 osd.3 since back 2024-09-17T07:29:04.077426+0000 front 2024-09-17T07:29:04.077361+0000 (oldest deadline 2024-09-17T07:29:28.777334+0000) 2024-09-17T07:29:30.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:30 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:30.451+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:43.250753+0000 front 2024-09-17T07:28:43.250667+0000 (oldest deadline 2024-09-17T07:29:06.150665+0000) 2024-09-17T07:29:30.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:30 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:30.451+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:51.352234+0000 front 2024-09-17T07:28:51.352171+0000 (oldest deadline 2024-09-17T07:29:17.252046+0000) 2024-09-17T07:29:30.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:30 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:30.451+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6822 osd.2 since back 2024-09-17T07:28:57.753273+0000 front 2024-09-17T07:28:57.753163+0000 (oldest deadline 2024-09-17T07:29:23.053104+0000) 2024-09-17T07:29:30.865 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:30 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:30.451+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6830 osd.3 since back 2024-09-17T07:29:04.754248+0000 front 2024-09-17T07:29:04.754234+0000 (oldest deadline 2024-09-17T07:29:29.454187+0000) 2024-09-17T07:29:31.733 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:31.421+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:43.250753+0000 front 2024-09-17T07:28:43.250667+0000 (oldest deadline 2024-09-17T07:29:06.150665+0000) 2024-09-17T07:29:31.734 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:31.421+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:51.352234+0000 front 2024-09-17T07:28:51.352171+0000 (oldest deadline 2024-09-17T07:29:17.252046+0000) 2024-09-17T07:29:31.734 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:31.421+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6822 osd.2 since back 2024-09-17T07:28:57.753273+0000 front 2024-09-17T07:28:57.753163+0000 (oldest deadline 2024-09-17T07:29:23.053104+0000) 2024-09-17T07:29:31.734 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:31 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:31.421+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6830 osd.3 since back 2024-09-17T07:29:04.754248+0000 front 2024-09-17T07:29:04.754234+0000 (oldest deadline 2024-09-17T07:29:29.454187+0000) 2024-09-17T07:29:32.055 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:31 smithi179 podman[80561]: 2024-09-17 07:29:31.733638294 +0000 UTC m=+5.438479392 container died 22baeb2f62cbca44f853c7ce856278e74fd3146bbd2bc472b1a1fe6740989d13 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-17T07:29:32.364 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:32 smithi179 podman[80561]: 2024-09-17 07:29:32.153486166 +0000 UTC m=+5.858327263 container cleanup 22baeb2f62cbca44f853c7ce856278e74fd3146bbd2bc472b1a1fe6740989d13 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6, ceph=True, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T07:29:32.364 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:32 smithi179 bash[80561]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6 2024-09-17T07:29:32.823 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:32 smithi179 podman[80576]: 2024-09-17 07:29:32.396394001 +0000 UTC m=+0.659208545 container remove 22baeb2f62cbca44f853c7ce856278e74fd3146bbd2bc472b1a1fe6740989d13 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True) 2024-09-17T07:29:32.824 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:32.388+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:43.250753+0000 front 2024-09-17T07:28:43.250667+0000 (oldest deadline 2024-09-17T07:29:06.150665+0000) 2024-09-17T07:29:32.824 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:32.388+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:51.352234+0000 front 2024-09-17T07:28:51.352171+0000 (oldest deadline 2024-09-17T07:29:17.252046+0000) 2024-09-17T07:29:32.824 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:32.388+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6822 osd.2 since back 2024-09-17T07:28:57.753273+0000 front 2024-09-17T07:28:57.753163+0000 (oldest deadline 2024-09-17T07:29:23.053104+0000) 2024-09-17T07:29:32.824 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:32 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:32.388+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6830 osd.3 since back 2024-09-17T07:29:04.754248+0000 front 2024-09-17T07:29:04.754234+0000 (oldest deadline 2024-09-17T07:29:29.454187+0000) 2024-09-17T07:29:33.114 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:32 smithi179 podman[80667]: 2024-09-17 07:29:32.726523745 +0000 UTC m=+0.016384102 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:29:33.114 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:32 smithi179 podman[80667]: 2024-09-17 07:29:32.847202719 +0000 UTC m=+0.137063110 container create a78dee7eb6ff09034daf4a3bdea19a7ce8668987344380ab3013b8a450e9b97b (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b) 2024-09-17T07:29:33.114 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:33 smithi179 podman[80667]: 2024-09-17 07:29:33.066631066 +0000 UTC m=+0.356491414 container init a78dee7eb6ff09034daf4a3bdea19a7ce8668987344380ab3013b8a450e9b97b (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T07:29:33.114 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:33 smithi179 podman[80667]: 2024-09-17 07:29:33.070492328 +0000 UTC m=+0.360352676 container start a78dee7eb6ff09034daf4a3bdea19a7ce8668987344380ab3013b8a450e9b97b (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-17T07:29:33.404 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:33 smithi179 podman[80667]: 2024-09-17 07:29:33.149684207 +0000 UTC m=+0.439544570 container attach a78dee7eb6ff09034daf4a3bdea19a7ce8668987344380ab3013b8a450e9b97b (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True) 2024-09-17T07:29:33.404 INFO:journalctl@ceph.osd.6.smithi179.stdout:Sep 17 07:29:33 smithi179 podman[80667]: 2024-09-17 07:29:33.249182637 +0000 UTC m=+0.539043003 container died a78dee7eb6ff09034daf4a3bdea19a7ce8668987344380ab3013b8a450e9b97b (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T07:29:33.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:33.403+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:43.250753+0000 front 2024-09-17T07:28:43.250667+0000 (oldest deadline 2024-09-17T07:29:06.150665+0000) 2024-09-17T07:29:33.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:33.403+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:51.352234+0000 front 2024-09-17T07:28:51.352171+0000 (oldest deadline 2024-09-17T07:29:17.252046+0000) 2024-09-17T07:29:33.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:33.403+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6822 osd.2 since back 2024-09-17T07:28:57.753273+0000 front 2024-09-17T07:28:57.753163+0000 (oldest deadline 2024-09-17T07:29:23.053104+0000) 2024-09-17T07:29:33.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:33 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:33.403+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6830 osd.3 since back 2024-09-17T07:29:04.754248+0000 front 2024-09-17T07:29:04.754234+0000 (oldest deadline 2024-09-17T07:29:29.454187+0000) 2024-09-17T07:29:33.896 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.6.service' 2024-09-17T07:29:33.934 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T07:29:33.934 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-17T07:29:33.934 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-17T07:29:33.935 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.7 2024-09-17T07:29:34.264 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:34 smithi179 systemd[1]: Stopping Ceph osd.7 for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:29:34.614 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:34.263+0000 7fd66ebba640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T07:29:34.614 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:34.263+0000 7fd66ebba640 -1 osd.7 136 *** Got signal Terminated *** 2024-09-17T07:29:34.614 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:34.263+0000 7fd66ebba640 -1 osd.7 136 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T07:29:34.614 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:34.434+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:43.250753+0000 front 2024-09-17T07:28:43.250667+0000 (oldest deadline 2024-09-17T07:29:06.150665+0000) 2024-09-17T07:29:34.614 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:34.434+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:51.352234+0000 front 2024-09-17T07:28:51.352171+0000 (oldest deadline 2024-09-17T07:29:17.252046+0000) 2024-09-17T07:29:34.614 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:34.434+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6822 osd.2 since back 2024-09-17T07:28:57.753273+0000 front 2024-09-17T07:28:57.753163+0000 (oldest deadline 2024-09-17T07:29:23.053104+0000) 2024-09-17T07:29:34.614 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:34 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:34.434+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6830 osd.3 since back 2024-09-17T07:29:04.754248+0000 front 2024-09-17T07:29:04.754234+0000 (oldest deadline 2024-09-17T07:29:29.454187+0000) 2024-09-17T07:29:35.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:35 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:35.387+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:43.250753+0000 front 2024-09-17T07:28:43.250667+0000 (oldest deadline 2024-09-17T07:29:06.150665+0000) 2024-09-17T07:29:35.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:35 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:35.387+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:51.352234+0000 front 2024-09-17T07:28:51.352171+0000 (oldest deadline 2024-09-17T07:29:17.252046+0000) 2024-09-17T07:29:35.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:35 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:35.387+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6822 osd.2 since back 2024-09-17T07:28:57.753273+0000 front 2024-09-17T07:28:57.753163+0000 (oldest deadline 2024-09-17T07:29:23.053104+0000) 2024-09-17T07:29:35.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:35 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:35.387+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6830 osd.3 since back 2024-09-17T07:29:04.754248+0000 front 2024-09-17T07:29:04.754234+0000 (oldest deadline 2024-09-17T07:29:29.454187+0000) 2024-09-17T07:29:36.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:36.396+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:43.250753+0000 front 2024-09-17T07:28:43.250667+0000 (oldest deadline 2024-09-17T07:29:06.150665+0000) 2024-09-17T07:29:36.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:36.396+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:51.352234+0000 front 2024-09-17T07:28:51.352171+0000 (oldest deadline 2024-09-17T07:29:17.252046+0000) 2024-09-17T07:29:36.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:36.396+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6822 osd.2 since back 2024-09-17T07:28:57.753273+0000 front 2024-09-17T07:28:57.753163+0000 (oldest deadline 2024-09-17T07:29:23.053104+0000) 2024-09-17T07:29:36.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:36 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:36.396+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6830 osd.3 since back 2024-09-17T07:29:04.754248+0000 front 2024-09-17T07:29:04.754234+0000 (oldest deadline 2024-09-17T07:29:29.454187+0000) 2024-09-17T07:29:37.765 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:37 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:37.444+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:43.250753+0000 front 2024-09-17T07:28:43.250667+0000 (oldest deadline 2024-09-17T07:29:06.150665+0000) 2024-09-17T07:29:37.765 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:37 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:37.444+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:51.352234+0000 front 2024-09-17T07:28:51.352171+0000 (oldest deadline 2024-09-17T07:29:17.252046+0000) 2024-09-17T07:29:37.765 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:37 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:37.444+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6822 osd.2 since back 2024-09-17T07:28:57.753273+0000 front 2024-09-17T07:28:57.753163+0000 (oldest deadline 2024-09-17T07:29:23.053104+0000) 2024-09-17T07:29:37.766 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:37 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:37.444+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6830 osd.3 since back 2024-09-17T07:29:04.754248+0000 front 2024-09-17T07:29:04.754234+0000 (oldest deadline 2024-09-17T07:29:29.454187+0000) 2024-09-17T07:29:38.782 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:38 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:38.406+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-09-17T07:28:43.250753+0000 front 2024-09-17T07:28:43.250667+0000 (oldest deadline 2024-09-17T07:29:06.150665+0000) 2024-09-17T07:29:38.782 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:38 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:38.406+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6814 osd.1 since back 2024-09-17T07:28:51.352234+0000 front 2024-09-17T07:28:51.352171+0000 (oldest deadline 2024-09-17T07:29:17.252046+0000) 2024-09-17T07:29:38.782 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:38 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:38.406+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6822 osd.2 since back 2024-09-17T07:28:57.753273+0000 front 2024-09-17T07:28:57.753163+0000 (oldest deadline 2024-09-17T07:29:23.053104+0000) 2024-09-17T07:29:38.782 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:38 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:38.406+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.44:6830 osd.3 since back 2024-09-17T07:29:04.754248+0000 front 2024-09-17T07:29:04.754234+0000 (oldest deadline 2024-09-17T07:29:29.454187+0000) 2024-09-17T07:29:38.782 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:38 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7[60701]: 2024-09-17T07:29:38.406+0000 7fd6671c6640 -1 osd.7 136 heartbeat_check: no reply from 172.21.15.179:6804 osd.4 since back 2024-09-17T07:29:11.754976+0000 front 2024-09-17T07:29:11.755013+0000 (oldest deadline 2024-09-17T07:29:37.655026+0000) 2024-09-17T07:29:39.864 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:39 smithi179 podman[80771]: 2024-09-17 07:29:39.599947629 +0000 UTC m=+5.437922437 container died 36b85c18e11f02852d35fcf1e085d24df28704b4172eecdf944e718f28941bb3 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-17T07:29:40.316 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:39 smithi179 podman[80771]: 2024-09-17 07:29:39.989241828 +0000 UTC m=+5.827216677 container cleanup 36b85c18e11f02852d35fcf1e085d24df28704b4172eecdf944e718f28941bb3 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T07:29:40.316 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:39 smithi179 bash[80771]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7 2024-09-17T07:29:40.614 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:40 smithi179 podman[80797]: 2024-09-17 07:29:40.316211247 +0000 UTC m=+0.711356940 container remove 36b85c18e11f02852d35fcf1e085d24df28704b4172eecdf944e718f28941bb3 (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T07:29:40.986 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:40 smithi179 podman[80886]: 2024-09-17 07:29:40.553283014 +0000 UTC m=+0.016085277 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9 2024-09-17T07:29:40.986 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:40 smithi179 podman[80886]: 2024-09-17 07:29:40.667147094 +0000 UTC m=+0.129949391 container create 6337b1338898cb2c26f840d39c3775e3bb0d984d10a9efe75a0c5cebf852662b (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS) 2024-09-17T07:29:40.986 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:40 smithi179 podman[80886]: 2024-09-17 07:29:40.8944668 +0000 UTC m=+0.357269060 container init 6337b1338898cb2c26f840d39c3775e3bb0d984d10a9efe75a0c5cebf852662b (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-17T07:29:40.987 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:40 smithi179 podman[80886]: 2024-09-17 07:29:40.897796255 +0000 UTC m=+0.360598511 container start 6337b1338898cb2c26f840d39c3775e3bb0d984d10a9efe75a0c5cebf852662b (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-17T07:29:41.364 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:40 smithi179 podman[80886]: 2024-09-17 07:29:40.986076536 +0000 UTC m=+0.448878796 container attach 6337b1338898cb2c26f840d39c3775e3bb0d984d10a9efe75a0c5cebf852662b (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T07:29:41.364 INFO:journalctl@ceph.osd.7.smithi179.stdout:Sep 17 07:29:41 smithi179 podman[80886]: 2024-09-17 07:29:41.088361832 +0000 UTC m=+0.551164084 container died 6337b1338898cb2c26f840d39c3775e3bb0d984d10a9efe75a0c5cebf852662b (image=quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T07:29:41.845 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-817ae996-74c3-11ef-bceb-c7b262605968@osd.7.service' 2024-09-17T07:29:41.889 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T07:29:41.889 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-17T07:29:41.889 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-09-17T07:29:41.890 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a 2024-09-17T07:29:42.114 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:29:41 smithi179 systemd[1]: Stopping Ceph prometheus.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:29:42.502 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:29:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:29:42.218Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-17T07:29:42.502 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:29:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:29:42.218Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-17T07:29:42.502 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:29:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:29:42.218Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-17T07:29:42.502 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:29:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:29:42.218Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-17T07:29:42.502 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:29:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:29:42.218Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-17T07:29:42.502 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:29:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:29:42.218Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-17T07:29:42.502 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:29:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:29:42.218Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-17T07:29:42.502 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:29:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:29:42.218Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-17T07:29:42.502 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:29:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:29:42.219Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-17T07:29:42.503 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:29:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:29:42.219Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-17T07:29:42.503 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:29:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:29:42.219Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-17T07:29:42.503 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:29:42 smithi179 ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a[72458]: ts=2024-09-17T07:29:42.219Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-17T07:29:42.503 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:29:42 smithi179 podman[80989]: 2024-09-17 07:29:42.228928313 +0000 UTC m=+0.113261080 container died 61e60ddd30e6b6f70e3296fd4da9b046fd4c7b9d726053bbf9c25ce72cd80da7 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T07:29:42.860 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:29:42 smithi179 podman[80989]: 2024-09-17 07:29:42.609548024 +0000 UTC m=+0.493880785 container cleanup 61e60ddd30e6b6f70e3296fd4da9b046fd4c7b9d726053bbf9c25ce72cd80da7 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T07:29:42.860 INFO:journalctl@ceph.prometheus.a.smithi179.stdout:Sep 17 07:29:42 smithi179 bash[80989]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-prometheus-a 2024-09-17T07:29:43.032 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-817ae996-74c3-11ef-bceb-c7b262605968@prometheus.a.service' 2024-09-17T07:29:43.066 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T07:29:43.066 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-09-17T07:29:43.066 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 817ae996-74c3-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-17T07:29:43.231 INFO:teuthology.orchestra.run.smithi044.stderr:usage: cephadm [-h] [--image IMAGE] [--docker] [--data-dir DATA_DIR] 2024-09-17T07:29:43.231 INFO:teuthology.orchestra.run.smithi044.stderr: [--log-dir LOG_DIR] [--logrotate-dir LOGROTATE_DIR] 2024-09-17T07:29:43.232 INFO:teuthology.orchestra.run.smithi044.stderr: [--unit-dir UNIT_DIR] [--verbose] [--timeout TIMEOUT] 2024-09-17T07:29:43.232 INFO:teuthology.orchestra.run.smithi044.stderr: [--retry RETRY] [--env ENV] [--no-container-init] 2024-09-17T07:29:43.235 INFO:teuthology.orchestra.run.smithi044.stderr: {version,pull,inspect-image,ls,list-networks,adopt,rm-daemon,rm-cluster,run,shell,enter,ceph-volume,unit,logs,bootstrap,deploy,check-host,prepare-host,add-repo,rm-repo,install,registry-login,gather-facts,exporter,host-maintenance,verify-prereqs} 2024-09-17T07:29:43.235 INFO:teuthology.orchestra.run.smithi044.stderr: ... 2024-09-17T07:29:43.235 INFO:teuthology.orchestra.run.smithi044.stderr:cephadm: error: unrecognized arguments: --keep-logs 2024-09-17T07:29:43.246 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-17T07:29:43.246 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 817ae996-74c3-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-17T07:29:43.390 INFO:teuthology.orchestra.run.smithi179.stderr:usage: cephadm [-h] [--image IMAGE] [--docker] [--data-dir DATA_DIR] 2024-09-17T07:29:43.390 INFO:teuthology.orchestra.run.smithi179.stderr: [--log-dir LOG_DIR] [--logrotate-dir LOGROTATE_DIR] 2024-09-17T07:29:43.390 INFO:teuthology.orchestra.run.smithi179.stderr: [--unit-dir UNIT_DIR] [--verbose] [--timeout TIMEOUT] 2024-09-17T07:29:43.390 INFO:teuthology.orchestra.run.smithi179.stderr: [--retry RETRY] [--env ENV] [--no-container-init] 2024-09-17T07:29:43.391 INFO:teuthology.orchestra.run.smithi179.stderr: {version,pull,inspect-image,ls,list-networks,adopt,rm-daemon,rm-cluster,run,shell,enter,ceph-volume,unit,logs,bootstrap,deploy,check-host,prepare-host,add-repo,rm-repo,install,registry-login,gather-facts,exporter,host-maintenance,verify-prereqs} 2024-09-17T07:29:43.391 INFO:teuthology.orchestra.run.smithi179.stderr: ... 2024-09-17T07:29:43.391 INFO:teuthology.orchestra.run.smithi179.stderr:cephadm: error: unrecognized arguments: --keep-logs 2024-09-17T07:29:43.404 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-17T07:29:43.405 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-17T07:29:43.431 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-17T07:29:43.456 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-17T07:29:43.458 DEBUG:teuthology.misc:Transferring archived files from smithi044:/var/lib/ceph/817ae996-74c3-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi/7908141/remote/smithi044/crash 2024-09-17T07:29:43.459 DEBUG:teuthology.orchestra.run.smithi044:> sudo tar c -f - -C /var/lib/ceph/817ae996-74c3-11ef-bceb-c7b262605968/crash -- . 2024-09-17T07:29:43.533 DEBUG:teuthology.misc:Transferring archived files from smithi179:/var/lib/ceph/817ae996-74c3-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi/7908141/remote/smithi179/crash 2024-09-17T07:29:43.534 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar c -f - -C /var/lib/ceph/817ae996-74c3-11ef-bceb-c7b262605968/crash -- . 2024-09-17T07:29:43.579 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-17T07:29:43.580 DEBUG:teuthology.orchestra.run.smithi044:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/817ae996-74c3-11ef-bceb-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v CEPHADM_STRAY_DAEMON | egrep -v CEPHADM_FAILED_DAEMON | egrep -v CEPHADM_AGENT_DOWN | head -n 1 2024-09-17T07:29:43.609 INFO:teuthology.orchestra.run.smithi044.stderr:grep: /var/log/ceph/817ae996-74c3-11ef-bceb-c7b262605968/ceph.log: No such file or directory 2024-09-17T07:29:43.611 INFO:tasks.cephadm:Compressing logs... 2024-09-17T07:29:43.611 DEBUG:teuthology.orchestra.run.smithi044:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T07:29:43.654 DEBUG:teuthology.orchestra.run.smithi179:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T07:29:43.677 INFO:teuthology.orchestra.run.smithi044.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-17T07:29:43.678 INFO:teuthology.orchestra.run.smithi179.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-17T07:29:51.163 INFO:tasks.cephadm:Archiving logs... 2024-09-17T07:29:51.164 DEBUG:teuthology.misc:Transferring archived files from smithi044:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi/7908141/remote/smithi044/log 2024-09-17T07:29:51.165 DEBUG:teuthology.orchestra.run.smithi044:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-17T07:29:52.036 DEBUG:teuthology.misc:Transferring archived files from smithi179:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi/7908141/remote/smithi179/log 2024-09-17T07:29:52.037 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-17T07:29:52.740 INFO:tasks.cephadm:Removing cluster... 2024-09-17T07:29:52.741 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 817ae996-74c3-11ef-bceb-c7b262605968 --force 2024-09-17T07:29:54.574 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:29:54 smithi044 systemd[1]: Stopping Ceph node-exporter.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:29:54.996 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:29:54 smithi044 podman[99524]: 2024-09-17 07:29:54.710405404 +0000 UTC m=+0.237237507 container died 15655da1de04d4bd939bb10394674a17bc62538fecf48a3caa145f6ce69f5763 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T07:29:55.325 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:29:55 smithi044 podman[99524]: 2024-09-17 07:29:55.195986043 +0000 UTC m=+0.722818207 container cleanup 15655da1de04d4bd939bb10394674a17bc62538fecf48a3caa145f6ce69f5763 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T07:29:55.325 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:29:55 smithi044 bash[99524]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a 2024-09-17T07:29:55.800 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:29:55 smithi044 podman[99537]: 2024-09-17 07:29:55.438949975 +0000 UTC m=+0.731586374 container remove 15655da1de04d4bd939bb10394674a17bc62538fecf48a3caa145f6ce69f5763 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T07:29:55.800 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:29:55 smithi044 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-17T07:29:55.800 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:29:55 smithi044 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-17T07:29:55.800 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:29:55 smithi044 systemd[1]: Stopped Ceph node-exporter.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:29:55.800 INFO:journalctl@ceph.node-exporter.a.smithi044.stdout:Sep 17 07:29:55 smithi044 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@node-exporter.a.service: Consumed 2.182s CPU time. 2024-09-17T07:29:56.074 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:29:55 smithi044 systemd[1]: Stopping Ceph alertmanager.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:29:56.074 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:29:56 smithi044 ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager-a[92359]: ts=2024-09-17T07:29:56.042Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-17T07:29:56.351 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:29:56 smithi044 podman[99645]: 2024-09-17 07:29:56.083425473 +0000 UTC m=+0.138577388 container died 6718ecf773210f90874722265d38d2dd85c385c8b6636ae5964fc8cd4ee9f21c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T07:29:56.729 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:29:56 smithi044 podman[99645]: 2024-09-17 07:29:56.45904514 +0000 UTC m=+0.514197046 container cleanup 6718ecf773210f90874722265d38d2dd85c385c8b6636ae5964fc8cd4ee9f21c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T07:29:56.729 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:29:56 smithi044 bash[99645]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager-a 2024-09-17T07:29:57.074 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:29:56 smithi044 podman[99657]: 2024-09-17 07:29:56.72682738 +0000 UTC m=+0.638108922 container remove 6718ecf773210f90874722265d38d2dd85c385c8b6636ae5964fc8cd4ee9f21c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T07:29:57.074 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:29:56 smithi044 podman[99657]: 2024-09-17 07:29:56.802412771 +0000 UTC m=+0.713694317 volume remove c6d1af4e1fe97800c61894d66ef5109babd59ef81d94a7bdeb9576e289ca2347 2024-09-17T07:29:57.074 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:29:56 smithi044 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-17T07:29:57.074 INFO:journalctl@ceph.alertmanager.a.smithi044.stdout:Sep 17 07:29:56 smithi044 systemd[1]: Stopped Ceph alertmanager.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:30:32.060 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 817ae996-74c3-11ef-bceb-c7b262605968 --force 2024-09-17T07:30:33.864 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:30:33 smithi179 systemd[1]: Stopping Ceph node-exporter.b for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:30:34.204 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:30:33 smithi179 podman[81492]: 2024-09-17 07:30:33.930093257 +0000 UTC m=+0.116687899 container died f768630b41ebed5a4cdfc764ffc4ec3d1f655bbbc9a2e35ef77a5f968a23e0d2 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T07:30:34.613 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:30:34 smithi179 podman[81492]: 2024-09-17 07:30:34.377575728 +0000 UTC m=+0.564170369 container cleanup f768630b41ebed5a4cdfc764ffc4ec3d1f655bbbc9a2e35ef77a5f968a23e0d2 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T07:30:34.614 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:30:34 smithi179 bash[81492]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b 2024-09-17T07:30:34.965 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:30:34 smithi179 podman[81506]: 2024-09-17 07:30:34.654096494 +0000 UTC m=+0.727437693 container remove f768630b41ebed5a4cdfc764ffc4ec3d1f655bbbc9a2e35ef77a5f968a23e0d2 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T07:30:34.965 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:30:34 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-17T07:30:34.965 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:30:34 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-17T07:30:34.965 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:30:34 smithi179 systemd[1]: Stopped Ceph node-exporter.b for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:30:34.965 INFO:journalctl@ceph.node-exporter.b.smithi179.stdout:Sep 17 07:30:34 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@node-exporter.b.service: Consumed 2.074s CPU time. 2024-09-17T07:30:35.218 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:30:34 smithi179 systemd[1]: Stopping Ceph grafana.a for 817ae996-74c3-11ef-bceb-c7b262605968... 2024-09-17T07:30:35.490 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:30:35 smithi179 podman[81614]: 2024-09-17 07:30:35.217831433 +0000 UTC m=+0.114774645 container died 65fcaac7b53fd4ceb79be0c207723440e3b2fa203642c5822b6e96aa3fb94ba1 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a, build-date=2022-04-29T11:48:11.055897, version=8.5, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , release=239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.component=ubi8-container, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, io.buildah.version=1.32.0) 2024-09-17T07:30:35.841 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:30:35 smithi179 podman[81614]: 2024-09-17 07:30:35.565008266 +0000 UTC m=+0.461951477 container cleanup 65fcaac7b53fd4ceb79be0c207723440e3b2fa203642c5822b6e96aa3fb94ba1 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, io.buildah.version=1.32.0, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-04-29T11:48:11.055897, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, vendor=Red Hat, Inc., description=Ceph Grafana Container, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.tags=base rhel8, release=239.1651231664) 2024-09-17T07:30:35.841 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:30:35 smithi179 bash[81614]: ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a 2024-09-17T07:30:36.114 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:30:35 smithi179 podman[81628]: 2024-09-17 07:30:35.841401762 +0000 UTC m=+0.620674146 container remove 65fcaac7b53fd4ceb79be0c207723440e3b2fa203642c5822b6e96aa3fb94ba1 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-817ae996-74c3-11ef-bceb-c7b262605968-grafana-a, io.buildah.version=1.32.0, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-type=git, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., build-date=2022-04-29T11:48:11.055897, description=Ceph Grafana Container, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, io.openshift.expose-services=, release=239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, name=ubi8, distribution-scope=public) 2024-09-17T07:30:36.114 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:30:35 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-09-17T07:30:36.114 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:30:35 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-09-17T07:30:36.114 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:30:35 smithi179 systemd[1]: Stopped Ceph grafana.a for 817ae996-74c3-11ef-bceb-c7b262605968. 2024-09-17T07:30:36.114 INFO:journalctl@ceph.grafana.a.smithi179.stdout:Sep 17 07:30:35 smithi179 systemd[1]: ceph-817ae996-74c3-11ef-bceb-c7b262605968@grafana.a.service: Consumed 4.214s CPU time. 2024-09-17T07:30:59.074 INFO:tasks.cephadm:Removing cephadm ... 2024-09-17T07:30:59.075 DEBUG:teuthology.orchestra.run.smithi044:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-17T07:30:59.097 DEBUG:teuthology.orchestra.run.smithi179:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-17T07:30:59.114 INFO:tasks.cephadm:Teardown complete 2024-09-17T07:30:59.115 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-17T07:30:59.125 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-17T07:30:59.125 DEBUG:teuthology.orchestra.run.smithi044:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T07:30:59.141 DEBUG:teuthology.orchestra.run.smithi179:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T07:30:59.156 INFO:teuthology.orchestra.run.smithi044.stderr:bash: line 1: ntpq: command not found 2024-09-17T07:30:59.160 INFO:teuthology.orchestra.run.smithi044.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T07:30:59.160 INFO:teuthology.orchestra.run.smithi044.stdout:=============================================================================== 2024-09-17T07:30:59.160 INFO:teuthology.orchestra.run.smithi044.stdout:^* hv01.front.sepia.ceph.com 3 6 377 45 +499us[ +521us] +/- 46ms 2024-09-17T07:30:59.160 INFO:teuthology.orchestra.run.smithi044.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 44 -4292us[-4292us] +/- 88ms 2024-09-17T07:30:59.160 INFO:teuthology.orchestra.run.smithi044.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 44 +2789us[+2789us] +/- 71ms 2024-09-17T07:30:59.160 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T07:30:59.171 INFO:teuthology.orchestra.run.smithi179.stderr:bash: line 1: ntpq: command not found 2024-09-17T07:30:59.175 INFO:teuthology.orchestra.run.smithi179.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T07:30:59.175 INFO:teuthology.orchestra.run.smithi179.stdout:=============================================================================== 2024-09-17T07:30:59.175 INFO:teuthology.orchestra.run.smithi179.stdout:^* hv01.front.sepia.ceph.com 3 6 377 43 +447us[ +565us] +/- 46ms 2024-09-17T07:30:59.175 INFO:teuthology.orchestra.run.smithi179.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 46 -4395us[-4278us] +/- 88ms 2024-09-17T07:30:59.175 INFO:teuthology.orchestra.run.smithi179.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 43 +2692us[+2810us] +/- 71ms 2024-09-17T07:30:59.175 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T07:30:59.176 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-17T07:30:59.186 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-17T07:30:59.186 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-17T07:30:59.197 DEBUG:teuthology.orchestra.run.smithi044:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-17T07:30:59.286 DEBUG:teuthology.orchestra.run.smithi179:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-17T07:30:59.360 DEBUG:teuthology.orchestra.run.smithi044:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T07:30:59.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:30:59.392 DEBUG:teuthology.orchestra.run.smithi179:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T07:30:59.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:30:59.421 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-17T07:30:59.429 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-17T07:30:59.438 INFO:teuthology.task.internal:Duration was 1636.024640 seconds 2024-09-17T07:30:59.438 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-17T07:30:59.446 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-17T07:30:59.446 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T07:30:59.449 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T07:30:59.487 INFO:teuthology.orchestra.run.smithi044.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T07:30:59.502 INFO:teuthology.orchestra.run.smithi179.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T07:30:59.851 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-17T07:30:59.851 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi044.front.sepia.ceph.com 2024-09-17T07:30:59.851 DEBUG:teuthology.orchestra.run.smithi044:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T07:30:59.915 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi179.front.sepia.ceph.com 2024-09-17T07:30:59.915 DEBUG:teuthology.orchestra.run.smithi179:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T07:30:59.938 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-17T07:30:59.938 DEBUG:teuthology.orchestra.run.smithi044:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T07:30:59.958 DEBUG:teuthology.orchestra.run.smithi179:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T07:31:00.004 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-17T07:31:00.005 DEBUG:teuthology.orchestra.run.smithi044:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T07:31:00.030 DEBUG:teuthology.orchestra.run.smithi179:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T07:31:00.653 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-17T07:31:00.662 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-17T07:31:00.662 DEBUG:teuthology.orchestra.run.smithi044:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T07:31:00.688 DEBUG:teuthology.orchestra.run.smithi179:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T07:31:00.714 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-17T07:31:00.723 DEBUG:teuthology.orchestra.run.smithi044:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T07:31:00.731 DEBUG:teuthology.orchestra.run.smithi179:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T07:31:00.752 INFO:teuthology.orchestra.run.smithi044.stdout:kernel.core_pattern = core 2024-09-17T07:31:00.778 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern = core 2024-09-17T07:31:00.826 DEBUG:teuthology.orchestra.run.smithi044:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T07:31:00.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:31:00.841 DEBUG:teuthology.orchestra.run.smithi179:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T07:31:00.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T07:31:00.881 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-17T07:31:00.891 INFO:teuthology.task.internal:Transferring archived files... 2024-09-17T07:31:00.892 DEBUG:teuthology.misc:Transferring archived files from smithi044:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi/7908141/remote/smithi044 2024-09-17T07:31:00.892 DEBUG:teuthology.orchestra.run.smithi044:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T07:31:00.938 DEBUG:teuthology.misc:Transferring archived files from smithi179:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi/7908141/remote/smithi179 2024-09-17T07:31:00.938 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T07:31:00.977 INFO:teuthology.task.internal:Removing archive directory... 2024-09-17T07:31:00.977 DEBUG:teuthology.orchestra.run.smithi044:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T07:31:00.979 DEBUG:teuthology.orchestra.run.smithi179:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T07:31:01.026 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-17T07:31:01.037 INFO:teuthology.task.internal:Not uploading archives. 2024-09-17T07:31:01.037 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-17T07:31:01.047 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-17T07:31:01.047 DEBUG:teuthology.orchestra.run.smithi044:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T07:31:01.049 DEBUG:teuthology.orchestra.run.smithi179:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T07:31:01.062 INFO:teuthology.orchestra.run.smithi044.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 07:31 /home/ubuntu/cephtest 2024-09-17T07:31:01.081 INFO:teuthology.orchestra.run.smithi179.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 07:31 /home/ubuntu/cephtest 2024-09-17T07:31:01.082 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-17T07:31:01.092 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-17T07:31:01.113 INFO:teuthology.run:Summary data: description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/classic} duration: 1636.024640083313 owner: scheduled_teuthology@teuthology success: true 2024-09-17T07:31:01.113 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T07:31:01.201 INFO:teuthology.run:pass